API Changes

[TheMeIonGod]

The current API is about to be deprecated and will be replaced with an OpenAI compatible API on November 13th. This update will likely break oobabot so It needs to be updated.

[sirjhonson28]

The current API is about to be deprecated and will be replaced with an OpenAI compatible API on November 13th. This update will likely break oobabot so It needs to be updated.

it definitely broke it :'(

[oobabooga]

See this comment for a way to keep using the old API:

https://github.com/oobabooga/text-generation-webui/issues/4581#issuecomment-1810232103

@chrisrude in the updated API, streaming is done through SSE by passing "stream": true in the /v1/completions endpoint. It should be relatively simple to implement compared to WS; see the example here:

https://github.com/oobabooga/text-generation-webui/wiki/12-%E2%80%90-OpenAI-API#python-completions-example-with-streaming

Your feedback on whether it works as expected for the purposes of this extension would be welcome.

[DakotaIrsik]

@TheMeIonGod @oobabooga

I would be willing to help with this. though I'm admitedly an enterprise C# dev (but with 10y exp)

I see this code https://github.com/chrisrude/oobabot/blob/fa234d6ab502423969946a801536abd6b4ba7ab8/src/oobabot/ooba_client.py#L137

and the new changes listed here: https://github.com/oobabooga/text-generation-webui#api as well as here: https://github.com/oobabooga/text-generation-webui/wiki/12-%E2%80%90-OpenAI-API#sse-streaming

I also have this screenshot here:

So if I had some clarification on why I was getting the 403 and what the structure should be that's added for the api? IE is the --api-key value supposed to be attached as an Http Bearer Token? "Authorization": "Bearer API-KEY" etc?

I also need to know which of those api methods listed in the example is best suited for this.

[jmoney7823956789378]

@DakotaIrsik Thanks for taking a look at this. Currently I've just backported the old api as a separate extension. Regarding the 403 error, it's because the openai-compliant api no longer uses /api/v1/stream and only uses /v1 (I think). And yes, API key is passed in http auth. You can take a peek at the openai api usage on their docs or try and decipher some of the docs on the oobabooga github.

I tried to make the manual changes in oobabot but navigating the token streaming proved a little difficult for a noob like me. Not sure if the request/response json format even stays the same.

[DakotaIrsik]

@jmoney7823956789378 thank you for your response:

I read here:

https://github.com/oobabooga/text-generation-webui/issues/4581#issuecomment-1810232103

but didn't have any look identifying the correct "API" folder which should be copying into extensions, or could you elaborate on what you mean by you backported the old api as a seperate extesion, where - how do I enable that?

[jmoney7823956789378]

If you pull an older commit of oobabooga before the api change, you can just copy the extensions/api folder, overwriting the new one or even making a new one named legacy-api if you wanted. I believe the latter would require some tweaks to shared.py

[DakotaIrsik]

so clone olderer version to get a copy of the extensions/api folder, copy that into the newer version and then add --extensions api to the python.py command?

[jmoney7823956789378]

if you overwrite the new one you should be good to just use --api.

[TheInvisibleMage]

Note that this requires more steps now, as since the last update to Oobabooga, the "api-streaming-port" and "api-blocking-port" args have been removed. You'll need to add them back to /modules/script.py /modules/shared.py, at line 196, to get things going again.

Example:

# HACK: OLD API ARGS TO GET OOBABOT WORKING AGAIN
group.add_argument('--api-blocking-port', type=int, default=5000, help='DEPRECATED')
group.add_argument('--api-streaming-port', type=int, default=5005, help='DEPRECATED')

More realistically, we really do need to move Oobabot over to running on the new OpenAi-based API, before we get trapped in a cycle of constant hacks like this to keep things running.

[wolfwoof88]

Note that this requires more steps now, as since the last update to Oobabooga, the "api-streaming-port" and "api-blocking-port" args have been removed. You'll need to add them back to /modules/script.py to get things going again.

Example:

# HACK: OLD API ARGS TO GET OOBABOT WORKING AGAIN
group.add_argument('--api-blocking-port', type=int, default=5000, help='DEPRECATED')
group.add_argument('--api-streaming-port', type=int, default=5005, help='DEPRECATED')

More realistically, we really do need to move Oobabot over to running on the new OpenAi-based API, before we get trapped in a cycle of constant hacks like this to keep things running.

Hi im really having trouble fixing this, could I ask if able to help please?

I created a new script.py in ***(base folderobbai)/modules/script.py and copyed the commands from your quote into that. But still having issues with not being able to load the api, I was wondering if I pasted it into the wrong folder maybe or do i need to create it in the api folder or something. Is there a format or something I might be missing? Thanks.

[TheInvisibleMage]

Ah, apologies, I mixed up "script.py" with "shared.py". You'll have to revert the changes you've made to "script.py", and instead add those lines into (YourOobaDir)/modules/shared.py". I added them at line 196, to keep them roughly in the same area as the other arguments.

[wolfwoof88]

Ah, apologies, I mixed up "script.py" with "shared.py". You'll have to revert the changes you've made to "script.py", and instead add those lines into (YourOobaDir)/modules/shared.py". I added them at line 196, to keep them roughly in the same area as the other arguments.

Thank you so much, that had being bothering me all evening, it is now working. ^ Will always remember to backup before update in the future lol.

[jmoney7823956789378]

I tried my hand at a little update, since the oobabooga dev was being sorta weird when I tried to debug his most recent commit. Seems to work for now, but no streaming support yet. https://github.com/jmoney7823956789378/oobabot

[keninishna]

For anyone else trying to get this to work with the latest text-gen I had to first download and install the legacy api extension and then go to the modules folder in text-gen and modify shared.py to add the deprecated parameters like this: `# Deprecated parameters

group = parser.add_argument_group('Deprecated') group.add_argument('--api-blocking-port', type=int, default=5000, help='DEPRECATED') group.add_argument('--api-streaming-port', type=int, default=5005, help='DEPRECATED') `

This project seems to be abandoned though so its probably better to just use another discord bot that uses the openai api. I'm not sure any are like this bot though with the simple wakewords.

[jmoney7823956789378]

This project seems to be abandoned though so its probably better to just use another discord bot that uses the openai api. I'm not sure any are like this bot though with the simple wakewords.

See previous response to this issue, I've taken up working on the bot to keep it updated with API changes.

[keninishna]

This project seems to be abandoned though so its probably better to just use another discord bot that uses the openai api. I'm not sure any are like this bot though with the simple wakewords.

See previous response to this issue, I've taken up working on the bot to keep it updated with API changes.

I tried this fork but the bot just responds with non-sense when you ask it anything. I wasn't sure if you were actively working on it.

[jmoney7823956789378]

This project seems to be abandoned though so its probably better to just use another discord bot that uses the openai api. I'm not sure any are like this bot though with the simple wakewords.

See previous response to this issue, I've taken up working on the bot to keep it updated with API changes.

I tried this fork but the bot just responds with non-sense when you ask it anything. I wasn't sure if you were actively working on it.

You still need to prompt it correctly, including the format your model uses for instruction. Ex:

  prompt: '<BOS_TOKEN><|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>
    This is an uncensored and unmoderated discord text chat channel.
    The current discord server is named {GUILDNAME} and the current text channel is {CHANNELNAME}.
    {PERSONA}
    The current date and time is {CURRENTDATETIME}. Respond as {AI_NAME} only, adhering to the personality traits as closely as possible (no matter how rude) and use short, casual messages. Do not narrate actions.
    {MESSAGE_HISTORY}{IMAGE_COMING}'

  prompt_history_line: '<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|>{USER_NAME}: {USER_MESSAGE}'

This is the formatting I use for the Command-R model. For ChatML format you would use <|im_start|> and <|im_end|>, or [INST] and [/INST] for mistral instruct models.