search

jmoney7823956789378 / oobabot

A Discord bot which talks to Large Language Model AIs running on oobabooga's text-generation-webui
MIT License
15 stars 3 forks source link

Is it possible to use the Superbooga extensions from Text-gen web UI in the API calls? #7

Closed AlanMW closed 5 months ago

AlanMW commented 5 months ago

I've noticed when using the API the generated responses don't use the extensions in Text generation web UI. I was trying to use Superbooga but text sent to the model didn't attempt to include RAG injected data where as chats in the Text generation web UI do.

This may be more of an issue with Text generation web UI, but figured I'd ask.

jmoney7823956789378 commented 5 months ago

I'll admit, I have long since stopped using oobabooga for textgen and moved to tabbyapi/aphrodite instead... But in the past I had the same issue. Extensions seem to interact very loosely with the api in most cases, and since superbooga doesnt keep stuff loaded outside the webui I'm not aware of any way to pass it into the api request/response. I might want to try looking into a "proxy" for rag, but i'm not quite skilled enough to pull that off yet.

AlanMW commented 5 months ago

Gotcha, and I assume you are just using this bot with one of those? Are you getting better performance from either of those or something? I'm not set on text Gen and if you are developing for something else I don't mind swapping.

jmoney7823956789378 commented 5 months ago

Are you getting better performance from either of those or something?

General consensus seems to be that loading models (and an API) without bothering with a webui is a slight bit faster. I'm not strictly developing for any specific API, either. In fact, I'm like 90% sure one of my most recent commits put support in for just about every API currently out, so don't worry about swapping unless you really need it's worth it. If you figure out the issue or a workaround for text-gen-webi's extensions/api problem, let me know :)

Currently verified API compatibility: text-generation-webui TabbyAPI llama-cpp-python Aphrodite (llama-cpp-python) Openrouter Cohere