search

henk717 / KoboldAI

KoboldAI is generative AI software optimized for fictional use, but capable of much more!
http://koboldai.com
GNU Affero General Public License v3.0
352 stars 130 forks source link

Feature request: Custom OpenAI API URL endpoint #484

Open DutchEllie opened 8 months ago

DutchEllie commented 8 months ago

Several services have started to make the OpenAI API standard their own. Not only if you (for some unknown reason) want to use Azure OpenAI Service, but most notably Oobabooga has recently migrated their API to the OpenAI standard.

Currently, the endpoint is fixed in code, but a simple edit should make it editable.

henk717 commented 8 months ago

Currently the priority of United has been local generations, so while nice to have we are currently focusing on improving / fixing the locally run backends instead of adding an online one.

For online backends I recommend using https://lite.koboldai.net which supports this and runs entirely in your browser. If you want something you can keep offline you can view and download its source here : https://github.com/LostRuins/lite.koboldai.net/blob/main/index.html

The existing OpenAI backend we have locally is so outdated it needs to be replaced entirely at this point, since its a much older standard that I think only GooseAI still uses at this point. So unfortunately not a simple edit to expose the field.

DutchEllie commented 8 months ago

Ah, okay. I guess that's why it didn't work when I just edited the field. I was trying to mess with the code to see if I could just shoot in a PR myself, seemed like a simple thing to fix. However, I see that it's also not working because of the outdated standard.

Well, I tried looking at the code myself to see if I could implement it somehow, but it's going way over my head as expected. Thanks for the recommendation of lite.koboldai.net. Maybe I'll try that or see if I can somehow load my GPTQ models from Ooba in your KoboldAI program instead.

henk717 commented 8 months ago

We support GPTQ out of the box, so all you have to do is move the folder over to our models folder. No need for ooba. Once you load from the folder you can change Huggingface to Exllama for extra speed.