Facing error when running API

[sablin39]

I've been using a modified AutoGPT which can define a custom openai_base_url. (The repo is https://github.com/DGdev91/Auto-GPT). However, when I set the base url to localhost:8000, which is the same as fastchat API, I get the following error. This the error on the AutoGPT side: openai.error.APIError: Invalid response object from API: 'Internal Server Error' (HTTP response code was 500)

This is the error on the API side:

 File "/home/rm2022/anaconda3/envs/llama/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout

I am wondering why this issue is caused and how to fix it. It would be better if I could keep the AutoGPT unchanged.

[merrymercy]

@andy-yang-1 Could you look into this?

[sablin39]

I've find out that the issue may be caused by asyncio.run() function, though the code looks the same as the one that works fine on stackoverflow. I am wondering if this is caused by the different python version (Which is 3.10 because I need to deploy AutoGPT)

[andy-yang-1]

I am also trying to deploy vicuna locally on auto-gpt, and I will try running your code to see where the issue lies. @sablin39

[andy-yang-1]

@sablin39 This issue might be caused by not changing the model name, as autogpt defaults to using gpt3.5-turbo. Can you successfully run it after changing the model name to vicuna-7b-v1.1?

[sablin39]

@sablin39 This issue might be caused by not changing the model name, as autogpt defaults to using gpt3.5-turbo. Can you successfully run it after changing the model name to vicuna-7b-v1.1?

Well...I've tried to change the model name to vicuna-13b-v1.1 in .env file of AutoGPT, but it caused the same issue. So I restored the .env file and tried to change the model name when starting the API. I've change to gpt-3.5-turbo in order to avoid potential conflict, and the result is shown on the screenshot above...

[andy-yang-1]

Can you provide your .env file and your running command? I want to reproduce the problem @sablin39

[sablin39]

Can you provide your .env file and your running command? I want to reproduce the problem @sablin39 I've only changed the following in .env file:

OPENAI_API_BASE_URL="http://localhost:8000"
EMBED_DIM=5120
USE_AZURE=False

The command I use is

python3 -m fastchat.serve.controller
python3 -m fastchat.serve.model_worker --model-name 'gpt-3.5-turbo' --model-path /path/to/vicuna/weights
export FASTCHAT_CONTROLLER_URL=http://localhost:21001
python3 -m fastchat.serve.api --host localhost --port 8000

Which is nearly the same as the commands shown in this repo. Moreover, the test shown in https://github.com/lm-sys/FastChat/blob/main/docs/openai_api.md#client-sdk works properly...

[andy-yang-1]

please change your .env file with:

FAST_TOKEN_LIMIT=2560

GPT-3.5 max token size is not equal to vicuna which caused this problem. However, directly connecting vicuna api with autogpt might cause a number of issues, and I feel that this won't be the only problem. I will release autogpt based on vicuna soon @sablin39

[sablin39]

please change your .env file with:

FAST_TOKEN_LIMIT=2560

GPT-3.5 max token size is not equal to vicuna which caused this problem. However, directly connecting vicuna api with autogpt might cause a number of issues, and I feel that this won't be the only problem. I will release autogpt based on vicuna soon @sablin39

I am trying to modify the autogpt from using openai api to using the client provided by fastchat. I am wondering if this will work.

[sablin39]

please change your .env file with:

FAST_TOKEN_LIMIT=2560

GPT-3.5 max token size is not equal to vicuna which caused this problem. However, directly connecting vicuna api with autogpt might cause a number of issues, and I feel that this won't be the only problem. I will release autogpt based on vicuna soon @sablin39

It seems that it works after changing some issues of tokens(which is changed into fixed numbers in the source code of AutoGPT).