Closed piercefreeman closed 3 months ago
This is not the first time this shows up, but the first time we have a report detailed enough and that we can work with. Thank you for putting the time, I will take a look!
Hi @piercefreeman
Thanks for your great debug details. This is an issue that we've run into a lot. It's not that the state machine has an infinite loop, it's that the language models suffer from a "Repetition Problem".
I ran your reproduction script with no error after changing
whitespace_pattern = None
to
whitespace_pattern=r"[ ]?"
.
@remic33 since this keeps coming up, we should consider setting the whitespace_pattern
default to [\t\r\n ]{0,8}
and allowing users to change it to None if they want to remove this reasonable constraint. I'll open a PR if you agree.
Should be solved by #916. Feel free to re-open if this problem persists.
Describe the issue as clearly as possible:
I saw some generation irregularities similar to https://github.com/outlines-dev/outlines/issues/450, so benchmarked it a bit further. It seems like there are certain classes of constraints that cause an infinite loop of the state machine, where the model keeps generating "valid" tokens but they don't contribute to the overall string content.
This script uses a pydantic schema that compiles to a valid regex and state machine, which has length constraints in place:
\{[\n ]*"poem"[\n ]*:[\n ]*"(?:[^"\\\x00-\x1f\x7f-\x9f]|\\.){150,200}"[\n ]*\}
. Each line of the sequence_generator is logged separately in the format:Within the loop, runs 0 and 1 complete successfully. On run 2 it stalls with the same maximum length output (
Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h
).Even as this generated text stays the same, the value.token_ids keep growing. The tokens evaluate to 28705 (
▁
) and 13 (<0x0A>
), over and over again:I've included a minimum reproducible case below.
Steps/code to reproduce the bug:
Expected result:
Error message:
Outlines/Python version information:
Version information:
Running on a H100 GPU.
Context for the issue:
No response