search

lm-sys / FastChat

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
Apache License 2.0
35.56k stars 4.37k forks source link

Using FastChat UI to Compare Multiple Local LLMs #3320

Open soufianechami opened 1 month ago

soufianechami commented 1 month ago

It's not clear in the repo readme how I can use the FastChat UI to compare multiple LLMs on my local machine.

I have these models served via FastAPIs and running on my local server.

Can anyone provide guidance or suggestions on how to set this up effectively?

jerilkuriakose commented 1 month ago

Step 1: Open a new terminal and launch the controller:

python3 -m fastchat.serve.controller

Step 2: Open a new terminal to launch the workers:

python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5

Note: Launch new terminals for each workers

Step 3: Open a new terminal to launch the web server:

python3 -m fastchat.serve.gradio_web_server_multi

The web server will give the URL.