Closed hudson-ai closed 3 weeks ago
So this passes, if instead of process_prompt([]) you do process_prompt([29871]) (29871 is " " token) Generally, if the model had a BOS you should pass [BOS] not []; phi-3 is kind of weird in that all strings start with space, so I guess you would pass plain space as BOS. This is not a problem is the prompt is not empty.
Hmm, I see... I'll have to revisit this. These tests were sort of a proxy for the fact that currently
lm + ("pi = " + gen("number", regex="[0-9]+"))
and
(lm + "pi = ") + gen("number", regex="[0-9]+")
have very different behaviors because the model sees different tokenizations.
Should be able to dig in a bit tomorrow
I may still need to rejigger this function a bit to pass the BOS token to process_prompt, but I was able to achieve the behavior I expected by calling the tokenizer's recode
function:
https://github.com/guidance-ai/guidance/pull/951/commits/cf181774423315ca861cf67d0e65a3ad8342a741
See the proper end to end test here: https://github.com/hudson-ai/guidance/blob/362278a20bc021dba3c94e0d156f13d3d32b77ec/tests/model_integration/test_model.py#L73
(copy this into your
test_ll.py
)If you can't get this to pass, I believe that it's acceptable to instead make this pass for