search

lm-sys / FastChat

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
Apache License 2.0
36.55k stars 4.51k forks source link

Models registered via web ui --register file.json don't appear as available on openai server list #3537

Open surak opened 1 day ago

surak commented 1 day ago

If you have a model loaded from a json file, it will appear on the list of the available models on the web UI, but it won't appear on the openai_api_server, because it's not added to the controller.

surak commented 1 day ago

Adding the model with the register_worker.py will fail with a 404.

For example, an ollama server is running at its default port 11434. If I add it via the json file, it opens fine on the web ui. If I try to open it with the python3 fastchat/serve/register_worker.py --controller http://localhost:21001 --worker-name http://localhost:11434, it will fail.

Should we add support for the json file on the register_worker.py or make the web interface add the worker to the controller?