chatbot in gradio server operates through json files, so gradio client cannot actually use remotely

pseudotensor commented 1 year ago

Describe the bug

This was my quick test to get chatbot through client working, but because it passes back files via API, it's useless remotely. So in later versions of the test code I just removed that code. But here it is in old state:

https://github.com/h2oai/h2ogpt/blob/05c742a1a283bf307bc745e8f276b0678f0ed9a9/client_test.py#L81-L107

Work-around is to just not support chatbot for client, which is disappointing.

Did I miss something?

Is there an existing issue for this?

[X] I have searched the existing issues

Reproduction

https://github.com/h2oai/h2ogpt/blob/05c742a1a283bf307bc745e8f276b0678f0ed9a9/client_test.py#L81-L107

Screenshot

No response

Logs

Not needed

System Info

gradio==3.27.0
gradio_client==0.1.3

Severity

serious, but I can work around it

abidlabs commented 1 year ago

Could you try setting serialize=False when you instantiate the Client? I think it should do what you need it to do:

https://github.com/gradio-app/gradio/blob/b66ecff6715be03030b3547c30d26dc3902f9094/client/python/gradio_client/client.py#LL66C15-L66C31

Just a head's up, I think this is still a little beta so signatures may change. Maybe we also need a better name for this parameter if you have any suggestions

pseudotensor commented 1 year ago

Ok, sorry if I missed that, will try it and report back.

pseudotensor commented 1 year ago

Hi @abidlabs . The problem with serialize=False is that I use a gr.State() as an input on server side. If I serialize=True, even my normal non-chatbot client calls work fine even though I'm only passing in all objects except the state.

However, if I use serialize=False, then even that basic test fails saying I'm missing an input, which is the state.

So forget about chatbot for now, just normal click behavior changes with serialize=True or False.

Here's the gradio server part:

https://github.com/h2oai/h2ogpt/blob/main/gradio_runner.py#L743-L746

See it has [model_state] as first entry.

Then see client code:

https://github.com/h2oai/h2ogpt/blob/main/client_test.py#L101-L106

See I'm using that API name, but not passing state.

In general, I cannot literally pass a state object (in general is a giant LLM) to the server from the client. The state should be something dynamic on the server yet no need to pass.

Thanks for any advice!

abidlabs commented 1 year ago

Hi @pseudotensor would you be able to try again setting serialize=False with the latest version of gradio on main? I believe the State issue has been fixed. You can install gradio from main by following the instructions here: https://gradio.app/docs/main

abidlabs commented 1 year ago

Hi @pseudotensor you should now be able to just test with gradio==3.29.0. I tested it here with this Space: https://huggingface.co/spaces/gradio/chatbot_simple

And you can see it working properly here:

In 4.0, we'll probably make this the default behavior of Chatbot with the Client (no need to pass in serialize=False)

I'll close this issue, but if you still run into problems, please free to reopen so we can figure it out.

pseudotensor commented 1 year ago

Hi @abidlabs , unfortunately that still does not work.

For https://github.com/h2oai/h2ogpt/pull/117/commits/48e24f195dc17982d6253449a02ecb8943135956 :

I updated to gradio 3.29.0 and client 0.2.2 and pass serialize=False for the chat case. It no longer asks for the State, but still shows a mismatch, now due to "chatbot" output (which is passed as input).

pytest -s -v test_client_calls::test_client_chat_nostream

gives:

Traceback (most recent call last):
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio/routes.py", line 408, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio/blocks.py", line 1313, in process_api
    inputs = self.preprocess_data(fn_index, inputs, state)
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio/blocks.py", line 1151, in preprocess_data
    self.validate_inputs(fn_index, inputs)
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio/blocks.py", line 1138, in validate_inputs
    raise ValueError(
ValueError: An event handler (fn) didn't receive enough input values (needed: 20, got: 19).
Check if the event handler calls a Javascript function, and make sure its return value is correct.
Wanted inputs:
    [textbox, textbox, textbox, checkbox, dropdown, slider, slider, slider, slider, slider, slider, checkbox, slider, slider, slider, checkbox, checkbox, textbox, textbox, chatbot]
Received inputs:
    ["Who are you?", "", "", False, "human_bot", 0.1, 0.75, 40, 1, 50, 0, False, 20, 1.0, 1, True, True, "", ""]
FAILED

If I try to add another "chatbot" entry as empty string (it's commented out right now) as:

    if chat:
        # add chatbot output on end.  Assumes serialize=False
        kwargs.update(dict(chatbot=''))

then I get:

Traceback (most recent call last):
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio/routes.py", line 408, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio/blocks.py", line 1313, in process_api
    inputs = self.preprocess_data(fn_index, inputs, state)
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio/blocks.py", line 1151, in preprocess_data
    self.validate_inputs(fn_index, inputs)
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio/blocks.py", line 1138, in validate_inputs
    raise ValueError(
ValueError: An event handler (bot) didn't receive enough input values (needed: 21, got: 20).
Check if the event handler calls a Javascript function, and make sure its return value is correct.
Wanted inputs:
    [textbox, textbox, textbox, checkbox, dropdown, slider, slider, slider, slider, slider, slider, checkbox, slider, slider, slider, checkbox, checkbox, textbox, textbox, state, chatbot]
Received inputs:
    ["Who are you?", "", "", False, "human_bot", 0.1, 0.75, 40, 1, 50, 0, False, 20, 1.0, 1, True, True, "", "", ""]
FAILED

i.e. it starts asking for the State() again.

pseudotensor commented 1 year ago

@abidlabs Note in your example: https://github.com/gradio-app/gradio/issues/4081#issuecomment-1540669112

You have no state, so it doesn't test that issue.

pseudotensor commented 1 year ago

However, this does work. So something is still different my full app that makes things not work:

import gradio as gr
import random
import time

def chatbot_with_state():
    with gr.Blocks() as demo:
        chatbot = gr.Chatbot()
        msg = gr.Textbox()
        clear = gr.Button("Clear")
        st = gr.State([1, 2, 3])

        def respond(st, message, chat_history):
            assert st[0] == 1 and st[1] == 2 and st[2] == 3
            bot_message = random.choice(["How are you?", "I love you", "I'm very hungry"])
            chat_history.append((message, bot_message))
            time.sleep(1)
            return "", chat_history

        msg.submit(respond, [st, msg, chatbot], [msg, chatbot], api_name="submit")
        clear.click(lambda: None, None, chatbot, queue=False)
        demo.launch(prevent_thread_lock=True)
    return demo

def chatbot_without_state():
    with gr.Blocks() as demo:
        chatbot = gr.Chatbot()
        msg = gr.Textbox()
        clear = gr.Button("Clear")

        def respond(message, chat_history):
            bot_message = random.choice(["How are you?", "I love you", "I'm very hungry"])
            chat_history.append((message, bot_message))
            time.sleep(1)
            return "", chat_history

        msg.submit(respond, [msg, chatbot], [msg, chatbot], api_name="submit")
        clear.click(lambda: None, None, chatbot, queue=False)
        demo.launch(prevent_thread_lock=True)
    return demo

if __name__ == "__main__":
    for chatbot_func in [chatbot_without_state, chatbot_with_state]:
        chatbot_func()
        print("chatbot %s" % chatbot_func.__name__)

        from gradio_client import Client

        client = Client("http://0.0.0.0:7860", serialize=False)

        ret = client.predict("Hello", [["", None]], api_name='/submit')
        print(ret)
        ret2 = client.predict("Hello", ret[1], api_name='/submit')
        print(ret2)

gives:

/home/jon/miniconda3/envs/h2ollm/bin/python3.10 /home/jon/h2o-llm/chatbot_simple_with_state.py
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
chatbot chatbot_without_state
Loaded as API: http://0.0.0.0:7860 ✔
('', [['', None], ['Hello', 'I love you']])
('', [['', None], ['Hello', 'I love you'], ['Hello', 'I’m very hungry']])
Running on local URL:  http://127.0.0.1:7861

To create a public link, set `share=True` in `launch()`.
chatbot chatbot_with_state
Loaded as API: http://0.0.0.0:7860 ✔
('', [['', None], ['Hello', 'How are you?']])
('', [['', None], ['Hello', 'How are you?'], ['Hello', 'I love you']])

Process finished with exit code 0

Above state case was invalid, I didn't close first demo, so 2nd demo was on 7681 and client was still trying to reach original demo without state. So the 2nd case above fails too if done properly like given later in this thread.

abidlabs commented 1 year ago

Do you have a code snippet that I can use to reproduce this issue (with both the gradio app and the client code)? Happy to investigate it some more on my end

pseudotensor commented 1 year ago

Hi, I'm trying to figure out enough to know what is a smaller repro. At moment it repros for that test I shared, but it would require you to install h2ogpt.

pseudotensor commented 1 year ago

Hi @abidlabs These state versions both fail. The .then one is just a bit closer to what I do, just in case some fix works for 2nd and not for .then:

import gradio as gr
import random
import time

def chatbot_with_state():
    with gr.Blocks() as demo:
        chatbot = gr.Chatbot()
        msg = gr.Textbox()
        clear = gr.Button("Clear")
        st = gr.State([1, 2, 3])

        def respond(message, st, chat_history):
            assert st[0] == 1 and st[1] == 2 and st[2] == 3
            bot_message = random.choice(["How are you?", "I love you", "I'm very hungry"])
            chat_history.append((message, bot_message))
            time.sleep(1)
            return "", chat_history

        msg.submit(respond, [msg, st, chatbot], [msg, chatbot], api_name="submit")
        clear.click(lambda: None, None, chatbot, queue=False)
        demo.queue()
        demo.launch(prevent_thread_lock=True)
    return demo

def chatbot_without_state():
    with gr.Blocks() as demo:
        chatbot = gr.Chatbot()
        msg = gr.Textbox()
        clear = gr.Button("Clear")

        def respond(message, chat_history):
            bot_message = random.choice(["How are you?", "I love you", "I'm very hungry"])
            chat_history.append((message, bot_message))
            time.sleep(1)
            return "", chat_history

        msg.submit(respond, [msg, chatbot], [msg, chatbot], api_name="submit")
        clear.click(lambda: None, None, chatbot, queue=False)
        demo.queue()
        demo.launch(prevent_thread_lock=True)
    return demo

def chatbot_with_state_then():
    with gr.Blocks() as demo:
        chatbot = gr.Chatbot()
        msg = gr.Textbox()
        clear = gr.Button("Clear")
        st = gr.State([1, 2, 3])

        def user(message, chat_history):
            return chat_history + [[message, None]]

        def respond(message, st, chat_history):
            assert st[0] == 1 and st[1] == 2 and st[2] == 3
            bot_message = random.choice(["How are you?", "I love you", "I'm very hungry"])
            chat_history.append((message, bot_message))
            time.sleep(1)
            return "", chat_history

        e1 = msg.submit(user, [msg, chatbot], chatbot, api_name="submit")
        e2 = e1.then(respond, [msg, st, chatbot], [msg, chatbot], api_name="submit_bot")
        clear.click(lambda: None, None, chatbot, queue=False)
        demo.queue()
        demo.launch(prevent_thread_lock=True)
    return demo

if __name__ == "__main__":
    for chatbot_func in [chatbot_without_state, chatbot_with_state]:
        demo = chatbot_func()
        print("chatbot %s" % chatbot_func.__name__)

        from gradio_client import Client

        client = Client("http://0.0.0.0:7860", serialize=False)

        chatbot = [["", None]]
        ret = client.predict("Hello", chatbot, api_name='/submit')
        print(ret)
        ret2 = client.predict("Hello", ret[1], api_name='/submit')
        print(ret2)

        demo.close()

    demo = chatbot_with_state_then()
    from gradio_client import Client

    client = Client("http://0.0.0.0:7860", serialize=False)

    chatbot = [["", None]]
    ret = client.predict("Hello", chatbot, api_name='/submit')
    print(ret)
    ret = client.predict("Hello", ret, api_name='/submit_bot')
    print(ret)

    demo.close()

Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
chatbot chatbot_without_state
Loaded as API: http://0.0.0.0:7860 ✔
('', [['', None], ['Hello', 'I love you']])
('', [['', None], ['Hello', 'I love you'], ['Hello', 'I’m very hungry']])
Closing server running on port: 7860
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
chatbot chatbot_with_state
Loaded as API: http://0.0.0.0:7860 ✔
Traceback (most recent call last):
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio/routes.py", line 408, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio/blocks.py", line 1313, in process_api
    inputs = self.preprocess_data(fn_index, inputs, state)
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio/blocks.py", line 1151, in preprocess_data
    self.validate_inputs(fn_index, inputs)
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio/blocks.py", line 1138, in validate_inputs
    raise ValueError(
ValueError: An event handler (respond) didn't receive enough input values (needed: 3, got: 2).
Check if the event handler calls a Javascript function, and make sure its return value is correct.
Wanted inputs:
    [textbox, state, chatbot]
Received inputs:
    ["Hello", [['', None]]]
Traceback (most recent call last):
  File "/home/jon/h2o-llm/chatbot_simple_with_state.py", line 80, in <module>
    ret = client.predict("Hello", [["", None]], api_name='/submit')
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio_client/client.py", line 263, in predict
    return self.submit(*args, api_name=api_name, fn_index=fn_index).result()
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio_client/client.py", line 937, in result
    raise self.future._exception  # type: ignore
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio_client/client.py", line 621, in _inner
    predictions = _predict(*data)
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio_client/client.py", line 653, in _predict
    raise ValueError(result["error"])
ValueError: None

Process finished with exit code 1

abidlabs commented 1 year ago

Ok taking a look soon

abidlabs commented 1 year ago

Thanks @pseudotensor for providing the helpful examples to reproduce.

Have a fix for this issue here: https://github.com/gradio-app/gradio/pull/4230. Feel free to try it out. Just one note -- for your chatbot_with_state_then() example, you'll need to call it via the Client like this.

client = Client(url, serialize=False)

chatbot = [["", None]]
ret = client.predict("Hello", chatbot, api_name='/submit')
print(ret)
ret = client.predict("Hello", ret[0], api_name='/submit_bot')
print(ret)

pseudotensor commented 1 year ago

Awesome, thanks. When is there a planned release with this fix?

abidlabs commented 1 year ago

Already out :) pip install --upgrade gradio-client

BinqiangLiu commented 11 months ago

Having various problems with Gradio API call errors: Most seen: File "/home/user/.local/lib/python3.10/site-packages/gradio_client/client.py", line 824, in _predict raise ValueError(result["error"]) ValueError: None

also often seen: on the API info page, no idea of how to deal with such arguments: "null", # str (filepath to JSON file) in 'parameter_4' Json component

also seen: File "/home/user/.local/lib/python3.10/site-packages/gradio_client/client.py", line 789, in _inner raise utils.InvalidAPIEndpointError() gradio_client.utils.InvalidAPIEndpointError

Would dear Gradio team post some really hands-on guidance on various use cases of Gradio API use on Huggingface?

Really look forward to that.

Thanks and regards.

gradio-app / gradio