search

ILikeAI / AlwaysReddy

AlwaysReddy is a LLM voice assistant that is always just a hotkey away.
MIT License
524 stars 50 forks source link

Openai compatible url for local completions? #39

Open Jake36921 opened 1 month ago

Jake36921 commented 1 month ago

lots of different backends i.e. koboldcpp has openai compatible url for their api

ultrmgns commented 1 month ago

Trying to install it the same way, no luck so far <3

aseichter2007 commented 2 weeks ago

The LMStudio config works against both koboldcpp and textgenwebui if you just change the url port to :5001(koboldcpp) or 5000(TGWUI)

I'm using it right now with koboldcpp, The only real difference is that koboldcpp uses "max_length" instead of "max_tokens"

It would be great to get a little love for open source solutions.

Koboldcpp now hosts a whisper endpoint as well.

Ollama is different in that it nests parameters.

My recommendation to the devs is support parameter control in the config like this: image note, those keys are just what my autocompletion suggested.