CarperAI / cheese

Used for adaptive human in the loop evaluation of language and embedding models.
MIT License
300 stars 24 forks source link

"Getting started" guide not working (no client id/password provided) #48

Closed asmith26 closed 1 year ago

asmith26 commented 1 year ago

I'm following https://cheese1.readthedocs.io/en/latest/started.html, but when I run the python command (related: https://github.com/CarperAI/cheese/pull/47) I get a URL but no client id/password:

python -m examples.image_selection
100%|██████ 1/1 [00:04<00:00,  4.97s/it]
~/miniconda3/envs/cheese/lib/python3.11/site-packages/gradio/components.py:206: UserWarning: 'rounded' styling is no longer supported. To round adjacent components together, place them in a Column(variant='box').
  warnings.warn(
~/miniconda3/envs/cheese/lib/python3.11/site-packages/gradio/components.py:224: UserWarning: 'border' styling is no longer supported. To place adjacent components in a shared border, place them in a Column(variant='box').
  warnings.warn(
Running on local URL:  http://127.0.0.1:7860

Desktop:

Many thanks for any help! :)

LouisCastricato commented 1 year ago

@shahbuland

shahbuland commented 1 year ago

You need to create a client. Create client is what gives you a user/pass, not launch. That being said the script should be doing that. I'll look into it

ayulockin commented 1 year ago

Hey @shahbuland I am also stuck on this page:

image
ayulockin commented 1 year ago

Okay I tried the docs_example.py example.

image

Turns out (1, 673696) is the (user id, password).

When trying this with image_selection.py I am getting this error though:

image

The image actually doesn't exist:

image

but ideally it should not crash, right?

asmith26 commented 1 year ago

Possibly another idea/quick question: is there a way to disable user authentication?


UPDATE/FOUND FIX: Removed share = True, from these lines:

https://github.com/CarperAI/cheese/blob/5148e80ee5850cb80cc9eb23c934d703fdf2502b/cheese/client/__init__.py#L300

https://github.com/CarperAI/cheese/blob/5148e80ee5850cb80cc9eb23c934d703fdf2502b/cheese/client/gradio_client.py#L276

ayulockin commented 1 year ago

Oh nice. Thanks @asmith26 :)

shahbuland commented 1 year ago

Possibly another idea/quick question: is there a way to disable user authentication?

UPDATE/FOUND FIX: Removed share = True, from these lines:

https://github.com/CarperAI/cheese/blob/5148e80ee5850cb80cc9eb23c934d703fdf2502b/cheese/client/__init__.py#L300

https://github.com/CarperAI/cheese/blob/5148e80ee5850cb80cc9eb23c934d703fdf2502b/cheese/client/gradio_client.py#L276

Bit confused by this. Share should just change whether the gradio demo is accessible by people outside your local connection. By disabling user auth, you are referring to the field where it asks for a user and password at start right? What would the desired behavior be (cc @ayulockin)? For it to just get you to the labelling screen immediately without any identifying information being needed? I see two options:

  1. Add option to drop user info entirely so that user ID is never kept/needed for anything. All you'd have to press is a start button to get started.
  2. Add option so that you do have to enter a user ID, but no password.

During development, I imagined that this imposed a security risk as it means anyone with url can start writing to your dataset. What is preferred?

asmith26 commented 1 year ago

Bit confused by this. Share should just change whether the gradio demo is accessible by people outside your local connection.

So I've found by removing share=True has fixed the actual problem in this issue (i.e. that I couldn't run the "Getting started" guide). I think what was happening without this change is that gradio would try to create such a public link/app, but I think a proxy network was blocking this, hence the app would just hang.

Maybe we could make this more customisable when cheese.launch() is ran (my personal preference/thought is to have share=False the default and then a user can use something like cheese.launch(share=True) as required).

By disabling user auth, you are referring to the field where it asks for a user and password at start right?

This was just an idea to help me get the "Getting started" example running - I'm less worried about this now that I have fixed my issue, but it might be beneficial to remove user auth if it's possible and simplify things etc.

djaym7 commented 1 year ago

You need to create a client. Create client is what gives you a user/pass, not launch. That being said the script should be doing that. I'll look into it

Which client to run and how ? Can you add that to readme ?

djaym7 commented 1 year ago

Figured it out, gotta add this. Is there a way to let users sign up ? or give predefined users and passowrds ? usr, passwd = cheese.create_client(1)

print(usr, passwd )