search

drewbaumann / AskGPT

GNU General Public License v3.0
44 stars 15 forks source link

Feature request #7

Open SmokeShine opened 10 months ago

SmokeShine commented 10 months ago

Is it possible to use llama openai compatibility ? https://github.com/ggerganov/llama.cpp/discussions/795

What changes would be required?

Topping1 commented 2 months ago

You can use my fork to use the KoboldCpp API. My fork: https://github.com/Topping1/AskKobold/tree/main KoboldCpp: https://github.com/LostRuins/koboldcpp

drewbaumann commented 1 month ago

Is it possible to use llama openai compatibility ? ggerganov/llama.cpp#795

What changes would be required?

Using @Topping1 's fork may be easiest, but forking and pointing to a server running a spec mimicking openai's isn't too challenging. I could imagine a preference option where one could pick their model or supply an openAI style compatible API endpoint of their own for local LLMs.