I seems to have deploy the server successfully, but a 404 error is returned in the web API

[Daeda1used]

Dear Jarvis Team

I'm new to posting issues on Github, Please excuse any unclear descriptions.

I followed the Guidance for server as follow:

# setup env
cd server
conda create -n jarvis python=3.8
conda activate jarvis
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
pip install -r requirements.txt

# download models
cd models
sh download.sh # required when `inference_mode` is `local` or `hybrid`

# run server
cd ..
python models_server.py --config config.yaml # required when `inference_mode` is `local` or `hybrid`
python awesome_chat.py --config config.yaml --mode server # for text-davinci-003

I believe I successfully download the models and I try run command: python models_server.py --config config.yaml I got followed return:

I change the config.yaml file as follow (mainly from localhost to 0.0.0.0 cause I run it on a server machine):

But when I try to test the model out according to the guidance, I got a 404 error as return, The guidance show me that I could send a request to the port to "communicate" with my deployed model on the server as follow:

# request
curl --location 'http://localhost:8004/tasks' \
--header 'Content-Type: application/json' \
--data '{
    "messages": [
        {
            "role": "user",
            "content": "based on pose of /examples/d.jpg and content of /examples/e.jpg, please show me a new image"
        }
    ]
}'

# response
[{"args":{"image":"/examples/d.jpg"},"dep":[-1],"id":0,"task":"openpose-control"},{"args":{"image":"/examples/e.jpg"},"dep":[-1],"id":1,"task":"image-to-text"},{"args":{"image":"<GENERATED>-0","text":"<GENERATED>-1"},"dep":[1,0],"id":2,"task":"openpose-text-to-image"}] 

So I write a small python code as:

import requests
import json

url = 'http://localhost:8005/hugginggpt'
headers = {'Content-Type': 'application/json'}
data = {
    "messages": [
        {
            "role": "user",
            "content": "please generate a video based on \"Spiderman is surfing\""
        }
    ]
}

response = requests.post(url, headers=headers, data=json.dumps(data))

if response.status_code == 200:
    result = json.loads(response.content)
    print(result)
else:
    print('Request failed with status code: %d' % response.status_code)

And I got following error:

Request failed with status code: 404

Similarly, I get the same message when I use my browser to access port 8005 of the ip of the server where I have deployed jarvis：

Is there anything I forget to do for a successfully deploy?

Many thanks!

[zzfoo]

Are you deploying on a remote server and requesting it locally? If so, I think you need to change the ip in the url to the remote server's ip and make sure the port is accessible

[Daeda1used]

Are you deploying on a remote server and requesting it locally? If so, I think you need to change the ip in the url to the remote server's ip and make sure the port is accessible

Hi, thanks for the kind reply.

I actually run the request code on my remote server itself, that's why I use "localhost" on my little request python code.

I also try to access my remote server port directly from my personal PC. Let's say my remote PC IP is 123.123.123.123, I type 123.123.123.123:8005 on my working PC, and get 404 again. I also checked my remote server ufw status:

Is there anything I need to modify?

Thanks.

[zzfoo]

Have you tried not change localhost to 0.0.0.0? I don't know, maybe it'll work.Also you can try run in cli mode or test mode python awesome_chat.py --config config.yaml --mode cli to make sure the model server is ok

[Daeda1used]

Have you tried not change localhost to 0.0.0.0? I don't know, maybe it'll work.Also you can try run in cli mode or test mode python awesome_chat.py --config config.yaml --mode cli to make sure the model server is ok

Sadly, even I change it back to localhost and re-run my request code above, it still shows the error 404.

And if I run command: python awesome_chat.py --config config.yaml --mode cli I got the error below:

Traceback (most recent call last):
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/home/yiming/mambaforge/envs/jarvis/lib/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/yiming/mambaforge/envs/jarvis/lib/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/yiming/mambaforge/envs/jarvis/lib/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/yiming/mambaforge/envs/jarvis/lib/python3.8/http/client.py", line 1011, in _send_output
    self.send(msg)
  File "/home/yiming/mambaforge/envs/jarvis/lib/python3.8/http/client.py", line 951, in send
    self.connect()
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fdff43b4130>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yiming/.local/lib/python3.8/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8005): Max retries exceeded with url: /running (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdff43b4130>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "awesome_chat.py", line 99, in <module>
    r = requests.get(Model_Server + "/running")
  File "/home/yiming/.local/lib/python3.8/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/home/yiming/.local/lib/python3.8/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/yiming/.local/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/yiming/.local/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/yiming/.local/lib/python3.8/site-packages/requests/adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8005): Max retries exceeded with url: /running (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdff43b4130>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "awesome_chat.py", line 103, in <module>
    raise ValueError(message)
ValueError: The server of local inference endpoints is not running, please start it first. (or using `inference_mode: huggingface` in config.yaml for a feature-limited experience)

Seems a connection refuse problem, I been working on this whole day, but I‘m not quite familiar with it. As I showing originally, my ufw status shows I allow connection via port 8003-8006, so I don't know if any thing I can Do with it.

[tricktreat]

Hi. According to the settings you provided, the host of the model server or the http server cannot be 0.0.0.0. Please try your LAN IP. In the next commit I will fix the issue to support 0.0.0.0 settings.

[Daeda1used]

Hi. According to the settings you provided, the host of the model server or the http server cannot be 0.0.0.0. Please try your LAN IP. In the next commit I will fix the issue to support 0.0.0.0 settings.

Hi @tricktreat

Thank you for helping me with it.

I changed it back to 'localhost' as default, re-run the command: python models_server.py --config config.yaml

By check the 'top' command on my remote server, I can see that around 64GB Memory has been used so the full model is deployed on my machine I assume.

I create a python file under 'server' directory, named 'localhost_test.py' and the code is like this:

import requests
import json

url = 'http://localhost:8005/hugginggpt'
headers = {'Content-Type': 'application/json'}
data = {
    "messages": [
        {
            "role": "user",
            "content": "please generate a video based on \"Spiderman is surfing\""
        }
    ]
}

response = requests.post(url, headers=headers, data=json.dumps(data))

if response.status_code == 200:
    result = json.loads(response.content)
    print(result)
else:
    print('Request failed with status code: %d' % response.status_code)

But still got:

Request failed with status code: 404

Process finished with exit code 0

[tricktreat]

Did you run this command? python awesome_chat.py --config config.yaml --mode server. And why you set url = 'http://localhost:8005/hugginggpt'. The /hugginggpt Web API is started by awesome_chat.py and the default port is 8004.

[Daeda1used]

An error has occurred as follow when I try to run awesome_chat.py:python awesome_chat.py --config config.yaml --mode server

Traceback (most recent call last):
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/home/yiming/mambaforge/envs/jarvis/lib/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/yiming/mambaforge/envs/jarvis/lib/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/yiming/mambaforge/envs/jarvis/lib/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/yiming/mambaforge/envs/jarvis/lib/python3.8/http/client.py", line 1011, in _send_output
    self.send(msg)
  File "/home/yiming/mambaforge/envs/jarvis/lib/python3.8/http/client.py", line 951, in send
    self.connect()
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f72cdb09130>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yiming/.local/lib/python3.8/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/home/yiming/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8005): Max retries exceeded with url: /running (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f72cdb09130>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "awesome_chat.py", line 99, in <module>
    r = requests.get(Model_Server + "/running")
  File "/home/yiming/.local/lib/python3.8/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/home/yiming/.local/lib/python3.8/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/yiming/.local/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/yiming/.local/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/yiming/.local/lib/python3.8/site-packages/requests/adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8005): Max retries exceeded with url: /running (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f72cdb09130>: Failed to establish a new connection: [Errno 111] Connection refused'))

[Daeda1used]

Update:

I make a really stupid mistake, but anyway, I successfully run command : python models_server.py --config config.yaml and then I ran: python awesome_chat.py --config config.yaml --mode server

No connection error again, but it Stuck on an error and didn't stop running automatically:

/home/yiming/mambaforge/envs/jarvis/lib/python3.8/site-packages/pydub/utils.py:170: RuntimeWarning: Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work
  warn("Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work", RuntimeWarning)
INFO:__main__:********************************************************************************
INFO:__main__:input: please generate a video based on "Spiderman is surfing"
DEBUG:__main__:[{'role': 'system', 'content': '#1 Task Planning Stage: The AI assistant can parse user input to several tasks: [{"task": task, "id", task_id, "dep": dependency_task_id, "args": {"text": text or <GENERATED>-dep_id, "image": image_url or <GENERATED>-dep_id,"audio": audio_url or <GENERATED>-dep_id}}]. The special tag "<GENERATED>-dep_id" refer to the one genereted text/image/audio in the dependency task (Please consider whether the dependency task generates resources of this type.) and "dep_id" must be in "dep" list. The "dep" field denotes the ids of the previous prerequisite tasks which generate a new resource that the current task relies on. The "args" field must in ["text", "image", "audio"], nothing else. The task MUST selected from the following options: "token-classification", "text2text-generation", "summarization", "translation",  "question-answering", "conversational", "text-generation", "sentence-similarity", "tabular-classification", "object-detection", "image-classification", "image-to-image", "image-to-text", "text-to-image", "text-to-video", visual-question-answering", "document-question-answering", "image-segmentation", "depth-estimation", "text-to-speech", "automatic-speech-recognition", "audio-to-audio", "audio-classification", "canny-control", "hed-control", "mlsd-control", "normal-control", "openpose-control", "canny-text-to-image", "depth-text-to-image", "hed-text-to-image", "mlsd-text-to-image", "normal-text-to-image", "openpose-text-to-image", "seg-text-to-image". There may be multiple tasks of the same type. Think step by step about all the tasks needed to resolve the user\'s request. Parse out as few tasks as possible while ensuring that the user request can be resolved. Pay attention to the dependencies and order among tasks. If the user input can\'t be parsed, you need reply empty JSON []. '}, {'role': 'user', 'content': 'Give you some pictures e1.jpg, e2.png, e3.jpg, help me count the number of sheep?'}, {'role': 'assistant', 'content': '[{"task": "image-to-text", "id": 0, "dep": [-1], "args": {"image": "e1.jpg" }}, {"task": "object-detection", "id": 1, "dep": [-1], "args": {"image": "e1.jpg" }}, {"task": "visual-question-answering", "id": 2, "dep": [1], "args": {"image": "<GENERATED>-1", "text": "How many sheep in the picture"}} }}, {"task": "image-to-text", "id": 3, "dep": [-1], "args": {"image": "e2.png" }}, {"task": "object-detection", "id": 4, "dep": [-1], "args": {"image": "e2.png" }}, {"task": "visual-question-answering", "id": 5, "dep": [4], "args": {"image": "<GENERATED>-4", "text": "How many sheep in the picture"}} }}, {"task": "image-to-text", "id": 6, "dep": [-1], "args": {"image": "e3.jpg" }},  {"task": "object-detection", "id": 7, "dep": [-1], "args": {"image": "e3.jpg" }}, {"task": "visual-question-answering", "id": 8, "dep": [7], "args": {"image": "<GENERATED>-7", "text": "How many sheep in the picture"}}]'}, {'role': 'user', 'content': 'Look at /e.jpg, can you tell me how many objects in the picture? Give me a picture and video similar to this one.'}, {'role': 'assistant', 'content': '[{"task": "image-to-text", "id": 0, "dep": [-1], "args": {"image": "/e.jpg" }}, {"task": "object-detection", "id": 1, "dep": [-1], "args": {"image": "/e.jpg" }}, {"task": "visual-question-answering", "id": 2, "dep": [1], "args": {"image": "<GENERATED>-1", "text": "how many objects in the picture?" }}, {"task": "text-to-image", "id": 3, "dep": [0], "args": {"text": "<GENERATED-0>" }}, {"task": "image-to-image", "id": 4, "dep": [-1], "args": {"image": "/e.jpg" }}, {"task": "text-to-video", "id": 5, "dep": [0], "args": {"text": "<GENERATED-0>" }}]'}, {'role': 'user', 'content': 'given a document /images/e.jpeg, answer me what is the student amount? And describe the image with your voice'}, {'role': 'assistant', 'content': '{"task": "document-question-answering", "id": 0, "dep": [-1], "args": {"image": "/images/e.jpeg", "text": "what is the student amount?" }}, {"task": "visual-question-answering", "id": 1, "dep": [-1], "args": {"image": "/images/e.jpeg", "text": "what is the student amount?" }}, {"task": "image-to-text", "id": 2, "dep": [-1], "args": {"image": "/images/e.jpg" }}, {"task": "text-to-speech", "id": 3, "dep": [2], "args": {"text": "<GENERATED>-2" }}]'}, {'role': 'user', 'content': 'Given an image /example.jpg, first generate a hed image, then based on the hed image generate a new image where a girl is reading a book'}, {'role': 'assistant', 'content': '[{"task": "openpose-control", "id": 0, "dep": [-1], "args": {"image": "/example.jpg" }},  {"task": "openpose-text-to-image", "id": 1, "dep": [0], "args": {"text": "a girl is reading a book", "image": "<GENERATED>-0" }}]'}, {'role': 'user', 'content': "please generate a video based on the text (prompt) 'a boy is running' and dub it"}, {'role': 'assistant', 'content': '[{"task": "text-to-video", "id": 0, "dep": [-1], "args": {"text": "a boy is running" }}, {"task": "text-to-speech", "id": 0, "dep": [-1], "args": {"text": "a boy is running" }}]'}, {'role': 'user', 'content': "The chat log [ [] ] may contain the resources I mentioned. Now I input { please generate a video based on 'Spiderman is surfing' }, please parse out as many as the required tasks to solve my request in a JSON format."}]
DEBUG:__main__:{
    "error": {
        "message": "You exceeded your current quota, please check your plan and billing details.",
        "type": "insufficient_quota",
        "param": null,
        "code": null
    }
}
ERROR:awesome_chat:Exception on /hugginggpt [POST]
Traceback (most recent call last):
  File "/home/yiming/mambaforge/envs/jarvis/lib/python3.8/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/yiming/mambaforge/envs/jarvis/lib/python3.8/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/yiming/mambaforge/envs/jarvis/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/yiming/mambaforge/envs/jarvis/lib/python3.8/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/yiming/mambaforge/envs/jarvis/lib/python3.8/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/home/yiming/Personal/JARVIS/server/awesome_chat.py", line 930, in chat
    response = chat_huggingface(messages)
  File "/home/yiming/Personal/JARVIS/server/awesome_chat.py", line 801, in chat_huggingface
    task_str = parse_task(context, input).strip()
  File "/home/yiming/Personal/JARVIS/server/awesome_chat.py", line 259, in parse_task
    return send_request(data)
  File "/home/yiming/Personal/JARVIS/server/awesome_chat.py", line 167, in send_request
    return response.json()["choices"][0]["text"].strip()
KeyError: 'choices'

Is there anything I can do? Thank you.

[tricktreat]

Please take note of this message: "You exceeded your current quota, please check your plan and billing details." Your quota has been reached.

[Daeda1used]

Please take note of this message: "You exceeded your current quota, please check your plan and billing details." Your quota has been reached.

Thanks a lot, everything make sense now!

Just a small question before closing it, where should I check my plan or billing to top up my quota? OpenAI api? or Hugging face?

Best.

[tricktreat]

OpenAI api

[Daeda1used]

Very glad to have @tricktreat here with me to help.