search

lm-sys / FastChat

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
Apache License 2.0
36.37k stars 4.47k forks source link

SillyTavern ui? FastChat compatible? #3265

Open QuinnPiers opened 4 months ago

QuinnPiers commented 4 months ago

Hi, sorry I'm a big newb at this.

I love that fastchat uses multi gpus and I've gotten the webservice/worker setup, but I don't know how to config SillyTavern to use fastchat and its not listed as one of the API connections options... https://docs.sillytavern.app/usage/api-connections/

but I wonder if Fastchat is the same as like Oobabooga? or 1 of the other options? https://docs.sillytavern.app/usage/api-connections/

QuinnPiers commented 4 months ago

https://github.com/SillyTavern/SillyTavern/blob/1c9b89fdcceee5ebadd17e9b34f23eb9f0b37eab/src/endpoints/chats.js#L41

QuinnPiers commented 4 months ago

Looks like fastchat has an openai interface, and sillytavern takes an openai interface... so it should work that way.

QuinnPiers commented 4 months ago

so in {fastChat,serve.controller}/docs I see that the endpoint isn't /v1/models its "/list_models" which makes me wonder if FastChat is openai compatible?