Open sablin39 opened 1 year ago
@andy-yang-1 Could you look into this?
I've find out that the issue may be caused by asyncio.run() function, though the code looks the same as the one that works fine on stackoverflow. I am wondering if this is caused by the different python version (Which is 3.10 because I need to deploy AutoGPT)
I am also trying to deploy vicuna locally on auto-gpt, and I will try running your code to see where the issue lies. @sablin39
@sablin39 This issue might be caused by not changing the model name, as autogpt defaults to using gpt3.5-turbo. Can you successfully run it after changing the model name to vicuna-7b-v1.1?
@sablin39 This issue might be caused by not changing the model name, as autogpt defaults to using gpt3.5-turbo. Can you successfully run it after changing the model name to vicuna-7b-v1.1?
Well...I've tried to change the model name to vicuna-13b-v1.1 in .env file of AutoGPT, but it caused the same issue. So I restored the .env
file and tried to change the model name when starting the API. I've change to gpt-3.5-turbo in order to avoid potential conflict, and the result is shown on the screenshot above...
Can you provide your .env
file and your running command? I want to reproduce the problem @sablin39
Can you provide your
.env
file and your running command? I want to reproduce the problem @sablin39 I've only changed the following in.env
file:OPENAI_API_BASE_URL="http://localhost:8000" EMBED_DIM=5120 USE_AZURE=False
The command I use is
python3 -m fastchat.serve.controller
python3 -m fastchat.serve.model_worker --model-name 'gpt-3.5-turbo' --model-path /path/to/vicuna/weights
export FASTCHAT_CONTROLLER_URL=http://localhost:21001
python3 -m fastchat.serve.api --host localhost --port 8000
Which is nearly the same as the commands shown in this repo. Moreover, the test shown in https://github.com/lm-sys/FastChat/blob/main/docs/openai_api.md#client-sdk works properly...
please change your .env
file with:
FAST_TOKEN_LIMIT=2560
GPT-3.5 max token size is not equal to vicuna which caused this problem. However, directly connecting vicuna api with autogpt might cause a number of issues, and I feel that this won't be the only problem. I will release autogpt based on vicuna soon @sablin39
please change your
.env
file with:FAST_TOKEN_LIMIT=2560
GPT-3.5 max token size is not equal to vicuna which caused this problem. However, directly connecting vicuna api with autogpt might cause a number of issues, and I feel that this won't be the only problem. I will release autogpt based on vicuna soon @sablin39
I am trying to modify the autogpt from using openai api to using the client
provided by fastchat. I am wondering if this will work.
please change your
.env
file with:FAST_TOKEN_LIMIT=2560
GPT-3.5 max token size is not equal to vicuna which caused this problem. However, directly connecting vicuna api with autogpt might cause a number of issues, and I feel that this won't be the only problem. I will release autogpt based on vicuna soon @sablin39
It seems that it works after changing some issues of tokens
(which is changed into fixed numbers in the source code of AutoGPT).
I've been using a modified AutoGPT which can define a custom openai_base_url. (The repo is https://github.com/DGdev91/Auto-GPT). However, when I set the base url to localhost:8000, which is the same as fastchat API, I get the following error. This the error on the AutoGPT side:
openai.error.APIError: Invalid response object from API: 'Internal Server Error' (HTTP response code was 500)
This is the error on the API side:
I am wondering why this issue is caused and how to fix it. It would be better if I could keep the AutoGPT unchanged.