Closed rojas-diego closed 2 months ago
Apologies, I didn't realise this was a duplicate of #1130. I see the suggested approach is to use streaming and drop the connection. Nonetheless, is it something that you would still consider?
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
Feature request
Add an optional
stop_regex
parameter toGenerateParameters
. Stop generating tokens ifstop_regex
matches the generated text.Motivation
Many applications such as inline code completion require fine-grained control over generated text. For example, you may want to generate a full "block" of code in which case you may desire to stop generation once a "dedent" is matched. Another example is generating till a matching a closing brace, quote or parenthesis is generated by the model. This is sometimes hard to express using TGI's "stop sequences" feature.
Your contribution
I have a working small demo, I'm open to making a PR if needed.
A few notes:
re
package can sometimes be very slow, especially in the presence of adversarial patterns. I think the best approach to this is to disable the feature by default (similar to how grammars work) and/or use a more restrictive Regex engine.