search

cyanff / anime.gf

Local & Open Source Alternative to CharacterAI
https://anime.gf
GNU Affero General Public License v3.0
425 stars 47 forks source link

Allow setting custom API endpoint #3

Closed AuroraNemoia closed 3 months ago

AuroraNemoia commented 3 months ago

Many local LLM servers such as llama.cpp's included server, and koboldcpp, offer an OpenAI-compatible server for the user's convenience. If we could change the endpoint URL for OpenAI, then it would support local inference right now.

cyanff commented 3 months ago

o7 will add

here's the flow I'm thinking of.

user goes to chat settings -> 
click on provider <select> field -> 
clicks "Custom" ->
instead of showing a model list, a URL field is shown to enter the endpoint

let me know if this makes sense

AuroraNemoia commented 3 months ago

I'd make sure to specify that you're expecting an OpenAI-compatible endpoint. But yeah, that sounds fine. That way, the OpenAI handler stays the same, and handles models the same.

mintisan commented 3 months ago

such as Ollama...

AuroraNemoia commented 3 months ago

ollama wraps llamacpp. All OpenAI compatible endpoints are the same. Why did you post this?

cyanff commented 3 months ago

released with v0.0.2! a bit of a naive implementation but I hope it works okay https://github.com/cyanff/anime.gf/releases/tag/0.0.2 feel free to reopen or ping me if there are issues :>