search

chrisrude / oobabot

A Discord bot which talks to Large Language Model AIs running on oobabooga's text-generation-webui
MIT License
101 stars 33 forks source link

API port 5000 / Gradio Live support #27

Open sidonsoft opened 1 year ago

sidonsoft commented 1 year ago

add non-streaming API support and gradio live using --public-api

chrisrude commented 1 year ago

I'm not sure I understand the request, could you please add a bit more context? What would the new behavior be?

sidonsoft commented 1 year ago

currently if you try to connect to the API on port 5000, the non-streaming API port, oobabot doesn't work. if you use --share --public-api the gradio live link also uses the non-streaming API port, so oobabot doesn't work

this would allow oobabot to connect to the API of a oobabooga server that doesn't have a publically accessible ip using gradio live

2023-05-18 06:49:16,466 INFO Oobabooga is at ws://2b79de1907fca22c21.gradio.live/ 2023-05-18 06:49:18,855 ERROR Could not connect to Oobabooga server: [ws://2b79de1907fca22c21.gradio.live/] 2023-05-18 06:49:18,855 ERROR Please check the URL and try again. 2023-05-18 06:49:18,855 ERROR Reason: 403, message='Invalid response status', url=URL('https://2b79de1907fca22c21.gradio.live:443/api/v1/stream')

chrisrude commented 1 year ago

Got it. This might be a good feature for someone to contribute to -- it would be pretty isolated to the ooba_client.py (plus adding the related settings). Anyone, let me know if you want tips on how to approach it!

Otherwise, I'd rank it later on the roadmap, just since I suspect most people are self-hosting their own Oobabooga server. But feel free to weigh in if this isn't the case for you, it would help prioritize this work over other features.