250 token limit on responses

[FellowChello]

Hey, I have changed the Max_New_Tokens to 400 everywhere I could find. Including the WebUi launch settings, But for some reason the bot continues to hit a hard limit of 250 tokens when responding.

Tested in the actual WebUi and had no issues generating responses up to the new limit of 400 tokens.

Is there any way to change this?

Thank you!

[FellowChello]

I think I found the solution: had to change AppData/Local/Programs/Python/Lb/Oobabot/ files to reflect the new token limit. After that it started to work, but broke the streamed responses.

can be closed. Thanks

[chrisrude]

Glad you got it to work! I'm curious what file you ended up needing to change.

In theory, you should be able to edit the following in the yaml configuration: oobabooga > request_params > max_new_tokens

If this doesn't work then there's still something to be fixed, so I just wanted to check.

[FellowChello]

I had to change %appdata%/Local/Programs/Python/Pthon/Python310/Lib/Site-packages/Oobabot/settings.py This is in a VM environment, so maybe that is the cause.

The file changes in Oobabot main directory had no effect, this was fresh install following the instructions. Weirdly enough I don't have the file you mentioned.

[jmoney7823956789378]

did you generate a config file? You (shouldn't) need to change anything in the actual code to get most things changed.

[FellowChello]

I hadn't used the config file. Created one now, but the config output shows that I had 400 token limit set for new_responses. Will try with config file and report back

[FellowChello]

And that seems to work, thank you! I changed the system files back to original values and the config file is working.

Is there a line in config file to increase the time bot waits before marking message as not received?

[chrisrude]

What do you mean "not received"?

The bot will wait indefinitely for a response from oobabooga, though it's possible that some other part of the HTTP chain will time out. But from our side we're unlimited.

When initially connecting to the service, we'll only wait for up to 5 seconds. But this is at startup, not when a message is received.

[jmoney7823956789378]

And that seems to work, thank you! I changed the system files back to original values and the config file is working.

Is there a line in config file to increase the time bot waits before marking message as not received?

I'm assuming you mean the time between a chat and when you send another message, without the wake-word or bot name included, and the bot doesn't respond? If you're into digging into python code, it should be manageable in src/oobabot/decide_to_respond.py (SOMEWHERE between line 104-162). If you use the --log-all-the-things option, you should get some more details on what the bot "sees" and the "chance" of a response.

[FellowChello]

Thank you for the clarification. After testing it for a day or two now everything seems to be stable even with new message limit of 500 thank you, this is great work!