Custom multiple stop token (for roleplay / conversation)

[wangerzi]

Based on role-playing needs, multiple stop token checks are implemented in example_flask.py

• Implemented multiple stop word definitions in generation
• Increased the complexity of the generate_simple function, but it is the best solution I can think of
• Does not affect execution efficiency when stop is not defined

No Stop Word:

Stop when "###end":

[turboderp]

I don't really want to keep overloading that one function with more and more features like this. E.g. this stop parameter would stop all sequences in a batch, and it's not guaranteed to catch all possible encodings of the given stop strings either.

But, I will be adding something like this probably later today, supporting multiple stop tokens and text strings as stop conditions, as well as streaming and context reuse. I think it will be a separate generator so as not to break too much.

[Kerushii]

I have updated example_ws to support custom stop token, oneshot as well as streaming and even token estimation. Please use ws if you can flask api only offers limited advantage when the resource is tight/needs to fulfil the request within its context. Most modern languages support ws. Additionally the ws example should be essentially faster than flask api due to cache reuse

[vadi2]

My 2c is that I'd like to see both http and websockets supported as first citizens, as http has an arguably lower bar to get started with.

[wangerzi]

My 2c is that I'd like to see both http and websockets supported as first citizens, as http has an arguably lower bar to get started with.

@vadi2 I agree with you, it's more useful in pre-testing phase.👍

As the author said, my code currently has a lot of problems, I will close this PR, and look forward to a better stop conditions implementation by turboderp.🆙🆙🆙

[turboderp]

Have a look at the latest commit, let me know if it's useful.

[vadi2]

At a scan, it looks really good! Thank you.