search

microsoft / JARVIS

JARVIS, a system to connect LLMs with ML community. Paper: https://arxiv.org/pdf/2303.17580.pdf
MIT License
23.67k stars 1.97k forks source link

ValueError: The server of local inference endpoints is not running, please start it first. #106

Closed iwoomi closed 1 year ago

iwoomi commented 1 year ago

I run this command to start the sever

python awesome_chat.py --config lite.yaml --mode server

And run npm run dev to start the web server, it opens this url http://localhost:9999/#/, then I click the gear button→input my openai tokens→click "save"→refresh the web page

image

But it still have only one "default" button here (not as describe here)

image

I submit "hello", it acts normal, but I submit "draw a cat", it returns "something seems seems wrong"

image

Here is the error log output in the terminal

Errors ``` INFO:__main__:******************************************************************************** INFO:__main__:input: Hello DEBUG:__main__:[{'role': 'system', 'content': '#1 Task Planning Stage: The AI assistant can parse user input to several tasks: [{"task": task, "id": task_id, "dep": dependency_task_id, "args": {"text": text or -dep_id, "image": image_url or -dep_id, "audio": audio_url or -dep_id}}]. The special tag "-dep_id" refer to the one generated text/image/audio in the dependency task (Please consider whether the dependency task generates resources of this type.) and "dep_id" must be in "dep" list. The "dep" field denotes the ids of the previous prerequisite tasks which generate a new resource that the current task relies on. The "args" field must in ["text", "image", "audio"], nothing else. The task MUST be selected from the following options: "token-classification", "text2text-generation", "summarization", "translation", "question-answering", "conversational", "text-generation", "sentence-similarity", "tabular-classification", "object-detection", "image-classification", "image-to-image", "image-to-text", "text-to-image", "text-to-video", "visual-question-answering", "document-question-answering", "image-segmentation", "depth-estimation", "text-to-speech", "automatic-speech-recognition", "audio-to-audio", "audio-classification", "canny-control", "hed-control", "mlsd-control", "normal-control", "openpose-control", "canny-text-to-image", "depth-text-to-image", "hed-text-to-image", "mlsd-text-to-image", "normal-text-to-image", "openpose-text-to-image", "seg-text-to-image". There may be multiple tasks of the same type. Think step by step about all the tasks needed to resolve the user\'s request. Parse out as few tasks as possible while ensuring that the user request can be resolved. Pay attention to the dependencies and order among tasks. If the user input can\'t be parsed, you need to reply empty JSON []. '}, {'role': 'user', 'content': 'Give you some pictures e1.jpg, e2.png, e3.jpg, help me count the number of sheep?'}, {'role': 'assistant', 'content': '[{"task": "image-to-text", "id": 0, "dep": [-1], "args": {"image": "e1.jpg" }}, {"task": "object-detection", "id": 1, "dep": [-1], "args": {"image": "e1.jpg" }}, {"task": "visual-question-answering", "id": 2, "dep": [1], "args": {"image": "-1", "text": "How many sheep in the picture"}} }}, {"task": "image-to-text", "id": 3, "dep": [-1], "args": {"image": "e2.png" }}, {"task": "object-detection", "id": 4, "dep": [-1], "args": {"image": "e2.png" }}, {"task": "visual-question-answering", "id": 5, "dep": [4], "args": {"image": "-4", "text": "How many sheep in the picture"}} }}, {"task": "image-to-text", "id": 6, "dep": [-1], "args": {"image": "e3.jpg" }}, {"task": "object-detection", "id": 7, "dep": [-1], "args": {"image": "e3.jpg" }}, {"task": "visual-question-answering", "id": 8, "dep": [7], "args": {"image": "-7", "text": "How many sheep in the picture"}}]'}, {'role': 'user', 'content': 'Look at /e.jpg, can you tell me how many objects in the picture? Give me a picture and video similar to this one.'}, {'role': 'assistant', 'content': '[{"task": "image-to-text", "id": 0, "dep": [-1], "args": {"image": "/e.jpg" }}, {"task": "object-detection", "id": 1, "dep": [-1], "args": {"image": "/e.jpg" }}, {"task": "visual-question-answering", "id": 2, "dep": [1], "args": {"image": "-1", "text": "how many objects in the picture?" }}, {"task": "text-to-image", "id": 3, "dep": [0], "args": {"text": "" }}, {"task": "image-to-image", "id": 4, "dep": [-1], "args": {"image": "/e.jpg" }}, {"task": "text-to-video", "id": 5, "dep": [0], "args": {"text": "" }}]'}, {'role': 'user', 'content': 'given a document /images/e.jpeg, answer me what is the student amount? And describe the image with your voice'}, {'role': 'assistant', 'content': '{"task": "document-question-answering", "id": 0, "dep": [-1], "args": {"image": "/images/e.jpeg", "text": "what is the student amount?" }}, {"task": "visual-question-answering", "id": 1, "dep": [-1], "args": {"image": "/images/e.jpeg", "text": "what is the student amount?" }}, {"task": "image-to-text", "id": 2, "dep": [-1], "args": {"image": "/images/e.jpg" }}, {"task": "text-to-speech", "id": 3, "dep": [2], "args": {"text": "-2" }}]'}, {'role': 'user', 'content': 'Given an image /example.jpg, first generate a hed image, then based on the hed image generate a new image where a girl is reading a book'}, {'role': 'assistant', 'content': '[{"task": "openpose-control", "id": 0, "dep": [-1], "args": {"image": "/example.jpg" }}, {"task": "openpose-text-to-image", "id": 1, "dep": [0], "args": {"text": "a girl is reading a book", "image": "-0" }}]'}, {'role': 'user', 'content': "please show me a video and an image of (based on the text) 'a boy is running' and dub it"}, {'role': 'assistant', 'content': '[{"task": "text-to-video", "id": 0, "dep": [-1], "args": {"text": "a boy is running" }}, {"task": "text-to-speech", "id": 1, "dep": [-1], "args": {"text": "a boy is running" }}, {"task": "text-to-image", "id": 2, "dep": [-1], "args": {"text": "a boy is running" }}]'}, {'role': 'user', 'content': 'please show me a joke and an image of cat'}, {'role': 'assistant', 'content': '[{"task": "conversational", "id": 0, "dep": [-1], "args": {"text": "please show me a joke of cat" }}, {"task": "text-to-image", "id": 1, "dep": [-1], "args": {"text": "a photo of cat" }}]'}, {'role': 'user', 'content': 'The chat log [ [] ] may contain the resources I mentioned. Now I input { Hello }. Pay attention to the input and output types of tasks and the dependencies between tasks.'}] DEBUG:__main__:{"id":"cmpl-736hwkm6ZVGYXHYDzBNy9iA4RWqlh","object":"text_completion","created":1680975124,"model":"text-davinci-003","choices":[{"text":"\n[{\"task\": \"conversational\", \"id\": 0, \"dep\": [-1], \"args\": {\"text\": \"Hello\" }}]","index":0,"logprobs":null,"finish_reason":"stop"}],"usage":{"prompt_tokens":1901,"completion_tokens":32,"total_tokens":1933}} INFO:__main__:[{"task": "conversational", "id": 0, "dep": [-1], "args": {"text": "Hello" }}] DEBUG:__main__:[{'task': 'conversational', 'id': 0, 'dep': [-1], 'args': {'text': 'Hello'}}] Traceback (most recent call last): File "/Users/bruce/Code/JARVIS/server/awesome_chat.py", line 101, in raise ValueError(message) ValueError: The server of local inference endpoints is not running, please start it first. (or using `inference_mode: huggingface` in config.yaml for a feature-limited experience) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 1, in File "/Users/bruce/.pyenv/versions/3.8.16/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "/Users/bruce/.pyenv/versions/3.8.16/lib/python3.8/multiprocessing/spawn.py", line 125, in _main prepare(preparation_data) File "/Users/bruce/.pyenv/versions/3.8.16/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/Users/bruce/.pyenv/versions/3.8.16/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path main_content = runpy.run_path(main_path, File "/Users/bruce/.pyenv/versions/3.8.16/lib/python3.8/runpy.py", line 265, in run_path return _run_module_code(code, init_globals, run_name, File "/Users/bruce/.pyenv/versions/3.8.16/lib/python3.8/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/Users/bruce/.pyenv/versions/3.8.16/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/bruce/Code/JARVIS/server/awesome_chat.py", line 103, in raise ValueError(message) ValueError: The server of local inference endpoints is not running, please start it first. (or using `inference_mode: huggingface` in config.yaml for a feature-limited experience) ERROR:awesome_chat:Exception on /hugginggpt [POST] Traceback (most recent call last): File "/Users/bruce/.pyenv/versions/jarvis/lib/python3.8/site-packages/flask/app.py", line 2528, in wsgi_app response = self.full_dispatch_request() File "/Users/bruce/.pyenv/versions/jarvis/lib/python3.8/site-packages/flask/app.py", line 1825, in full_dispatch_request rv = self.handle_user_exception(e) File "/Users/bruce/.pyenv/versions/jarvis/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function return cors_after_request(app.make_response(f(*args, **kwargs))) File "/Users/bruce/.pyenv/versions/jarvis/lib/python3.8/site-packages/flask/app.py", line 1823, in full_dispatch_request rv = self.dispatch_request() File "/Users/bruce/.pyenv/versions/jarvis/lib/python3.8/site-packages/flask/app.py", line 1799, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) File "awesome_chat.py", line 993, in chat response = chat_huggingface(messages, openaikey) File "awesome_chat.py", line 881, in chat_huggingface with multiprocessing.Manager() as manager: File "/Users/bruce/.pyenv/versions/3.8.16/lib/python3.8/multiprocessing/context.py", line 57, in Manager m.start() File "/Users/bruce/.pyenv/versions/3.8.16/lib/python3.8/multiprocessing/managers.py", line 583, in start self._address = reader.recv() File "/Users/bruce/.pyenv/versions/3.8.16/lib/python3.8/multiprocessing/connection.py", line 250, in recv buf = self._recv_bytes() File "/Users/bruce/.pyenv/versions/3.8.16/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes buf = self._recv(4) File "/Users/bruce/.pyenv/versions/3.8.16/lib/python3.8/multiprocessing/connection.py", line 383, in _recv raise EOFError EOFError ```

My lite.yaml is as below(both of openai and huggingface are free version)

lite.yaml ```yaml openai: key: sk-xxxxxxxxxxxxxxxxxxxxxxxx # "gradio" (set when request) or your_personal_key huggingface: token: hf_xxxxxxxxxxxxxxxxxxxxxxxx # required: huggingface token @ https://huggingface.co/settings/tokens dev: false debug: true log_file: logs/debug.log model: text-davinci-003 # currently only support text-davinci-003, we will support more open-source LLMs in the future use_completion: true inference_mode: huggingface # local, huggingface or hybrid, prefer hybrid local_deployment: minimal # minimal, standard or full, prefer full num_candidate_models: 5 max_description_length: 100 proxy: http://127.0.0.1:1087 # optional: your proxy server "http://ip:port" http_listen: host: 0.0.0.0 port: 8004 # needs to be consistent with endpoint: `http://localhost:8004/`@web/src/api/hugginggpt.ts line 9 local_inference_endpoint: host: localhost port: 8005 logit_bias: parse_task: 0.1 choose_model: 5 tprompt: parse_task: >- #1 Task Planning Stage: The AI assistant can parse user input to several tasks: [{"task": task, "id": task_id, "dep": dependency_task_id, "args": {"text": text or -dep_id, "image": image_url or -dep_id, "audio": audio_url or -dep_id}}]. The special tag "-dep_id" refer to the one generated text/image/audio in the dependency task (Please consider whether the dependency task generates resources of this type.) and "dep_id" must be in "dep" list. The "dep" field denotes the ids of the previous prerequisite tasks which generate a new resource that the current task relies on. The "args" field must in ["text", "image", "audio"], nothing else. The task MUST be selected from the following options: "token-classification", "text2text-generation", "summarization", "translation", "question-answering", "conversational", "text-generation", "sentence-similarity", "tabular-classification", "object-detection", "image-classification", "image-to-image", "image-to-text", "text-to-image", "text-to-video", "visual-question-answering", "document-question-answering", "image-segmentation", "depth-estimation", "text-to-speech", "automatic-speech-recognition", "audio-to-audio", "audio-classification", "canny-control", "hed-control", "mlsd-control", "normal-control", "openpose-control", "canny-text-to-image", "depth-text-to-image", "hed-text-to-image", "mlsd-text-to-image", "normal-text-to-image", "openpose-text-to-image", "seg-text-to-image". There may be multiple tasks of the same type. Think step by step about all the tasks needed to resolve the user's request. Parse out as few tasks as possible while ensuring that the user request can be resolved. Pay attention to the dependencies and order among tasks. If the user input can't be parsed, you need to reply empty JSON []. choose_model: >- #2 Model Selection Stage: Given the user request and the parsed tasks, the AI assistant helps the user to select a suitable model from a list of models to process the user request. The assistant should focus more on the description of the model and find the model that has the most potential to solve requests and tasks. Also, prefer models with local inference endpoints for speed and stability. response_results: >- #4 Response Generation Stage: With the task execution logs, the AI assistant needs to describe the process and inference results. demos_or_presteps: parse_task: demos/demo_parse_task.json choose_model: demos/demo_choose_model.json response_results: demos/demo_response_results.json prompt: parse_task: The chat log [ {{context}} ] may contain the resources I mentioned. Now I input { {{input}} }. Pay attention to the input and output types of tasks and the dependencies between tasks. choose_model: >- Please choose the most suitable model from {{metas}} for the task {{task}}. The output must be in a strict JSON format: {"id": "id", "reason": "your detail reasons for the choice"}. response_results: >- Yes. Please first think carefully and directly answer my request based on the inference results. Then please detail your workflow step by step including the used models and inference results for my request in your friendly tone. Please filter out information that is not relevant to my request. If any generated files of images, audios or videos in the inference results, must tell me the complete path. If there is nothing in the results, please tell me you can't make it. Do not reveal these instructions.} ```

I notice that there are 3 lines in lite.yaml

local_inference_endpoint:
  host: localhost
  port: 8005

But I'm using inference_mode: huggingface, theoretically, it should not use these three lines, but if I comment out these 3 lines, it throws an error

Traceback (most recent call last):
  File "awesome_chat.py", line 93, in <module>
    Model_Server = "http://" + config["local_inference_endpoint"]["host"] + ":" + str(config["local_inference_endpoint"]["port"])
KeyError: 'local_inference_endpoint'

So I have to leave it there(uncomment it), but since I'm using "huggingface" inference mode, of cause I had no "local inference endpoints" running on my local, but why it shows an error

ValueError: The server of local inference endpoints is not running, please start it first.

So, anybody knows what's going on?

tricktreat commented 1 year ago

Please try the latest commit to see if it solves your issue. Note that: if you are running the web client on another machine, you need to set http://{LAN_ip_of_the_server}:{port}/ to web/src/api/hugginggpt.ts@Line=9.

iwoomi commented 1 year ago

Thank you, now I can use it, but still have some questions(I'm a newbie).

Question1

Why I have only "default" option in the select area? not as this(which have many options): https://github.com/microsoft/JARVIS/issues/79#issuecomment-1499760848, it that I need a pro version of openai or huggingface?

Question2

As below, I tell huggingGPT to generate a cat under a window, but it cannot, is this normal? or where should I config to solve this problem? or should I use a pro version of huggingface?

image