Open RobinPicard opened 1 month ago
I've been looking into this issue.
I'm not sure why, but it appears that transformers with beam search calls the logits processor with a sequence it didn't actually sample only for the last token, then throws away the last sequence without impacting generation. Is there something I'm missing?
Here's an example with a modified pattern requiring 3 @
['1', '2', '3']
['1', '2', '3']
['1', '2', '3', 'age']
['1', '2', '3', 'ers']
['1', '2', '3', 'age', '@']
['1', '2', '3', 'ers', '@']
['1', '2', '3', 'age', '@', '@']
['1', '2', '3', 'ers', '@', '@']
['1', '2', '3', 'age', '@', '@', '@']
['1', '2', '3', 'ers', '@', '@', '@']
['1', '2', '3', 'age', '@', '@', '@', '!']
['1', '2', '3', 'age', '@', '@', '@', '#']
age@@@
ers@@@
With 4 samples:
['1', '2', '3', 'age', '@', '@', '@']
['1', '2', '3', 'ag', 'n', '@', '@']
['1', '2', '3', 'ag', 'c', '@', '@']
['1', '2', '3', 'ag', 'a', '@', '@']
['1', '2', '3', 'ag', 'n', '@', '@', '@']
['1', '2', '3', 'ag', 'c', '@', '@', '@']
['1', '2', '3', 'ag', 'a', '@', '@', '@']
['1', '2', '3', 'age', '@', '@', '@', '#']
['1', '2', '3', 'ag', 'n', '@', '@', '@', "'"]
['1', '2', '3', 'ag', 'n', '@', '@', '@', '#']
['1', '2', '3', 'ag', 'n', '@', '@', '@', '!']
['1', '2', '3', 'ag', 'n', '@', '@', '@', '$']
age@@@
agn@@@
agc@@@
aga@@@
Describe the issue as clearly as possible:
Encountered while working on PR 531
When generating several samples for a prompt with the transformers model, the
input_ids
gets modified during generation without affecting the final result. This causes problems in theFSMLogitsProcessor.process_logits
method as the keys ofself._fsm_states
depend on the values ofinput_ids
. This is easier to understand by just looking at the example below.I suppose it's related to "- Ensure
FSMLogitsProcessor
allows unstable sequence ordering (beam search in transformers and vLLM change the order of sequences)" mentioned in this commit by @lapp0, but here it's not just a change of order between the 1st and the 2nd sequence.Steps/code to reproduce the bug:
At the beginning of
OutlinesLogitsProcessor.__call__
, add:Expected result:
Not have 320 turn into 491 and 'ers' turn into 'age' at the 4th step of the generation for the 2nd sample
Error message:
Outlines/Python version information:
Current main branch (latest commit 9ce0df3)
Context for the issue:
No response