search

anon998 / simple-proxy-for-tavern

GNU Affero General Public License v3.0
110 stars 6 forks source link

Cannot use oobabooga/text-generation-webui as a backend #25

Open BenJackGill opened 1 year ago

BenJackGill commented 1 year ago

I am trying to stream from a runpod.io sever that is hosting oobabooga.

I updated config.mjs like this (using a fake IP here but you get the idea):

oobaStreamUrl: "ws://123.123.1.123:12345/api/v1/stream",

But it gives this error when connecting:

GET /v1/models
Error: Couldn't connect with a Kobold/KoboldCPP/Ooba backend.
request to http://127.0.0.1:5001/api/extra/version failed, reason: connect ECONNREFUSED 127.0.0.1:5001
request to http://127.0.0.1:10000/v1/models failed, reason: connect ECONNREFUSED 127.0.0.1:10000
request to http://127.0.0.1:8080/ failed, reason: connect ECONNREFUSED 127.0.0.1:8080
    at getBackendType (file:///Users/BenJackGill/Dev/ai/simple-proxy-for-tavern/src/index.mjs:201:9)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async checkWhichBackend (file:///Users/BenJackGill/Dev/ai/simple-proxy-for-tavern/src/index.mjs:206:45)
    at async Server.<anonymous> (file:///Users/BenJackGill/Dev/ai/simple-proxy-for-tavern/src/index.mjs:544:5)

Judging from that message, it never even tries to stream from Ooba. All of those attempts are for other backends. Ooba is not there.

I also tried changing backendType: null to backendType: "ooba" but it had no effect.

So how do I use ooba as a backend?

Janne2211 commented 1 year ago

you also have to fill in kobaldApiUrl . If you are using Ooba in Runpod then use the link from the url from "HTTP Service [port 5000]" within 'Connection Options'

BenJackGill commented 1 year ago

Thanks for the help.

After updating config.mjs with this:

koboldApiUrl: "https://abcde-12345.proxy.runpod.net/",
oobaStreamUrl: "ws://123.123.1.123:12345/api/v1/stream",

I can now connect, but I get this error in simple-proxy-for-tavern whenever I try to chat:

TypeError: messages.findLast is not a function
    at addMetadataToMessages (file:///Users/BenJackGill/Dev/ai/simple-proxy-for-tavern/src/parse-messages.mjs:210:24)
    at parseMessages (file:///Users/BenJackGill/Dev/ai/simple-proxy-for-tavern/src/parse-messages.mjs:237:21)
    at getChatCompletions (file:///Users/BenJackGill/Dev/ai/simple-proxy-for-tavern/src/index.mjs:417:56)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Server.<anonymous> (file:///Users/BenJackGill/Dev/ai/simple-proxy-for-tavern/src/index.mjs:561:9)

And also this error in Silly Tavern itself:

Error: Request failed with status code 501