search

guidance-ai / guidance

A guidance language for controlling large language models.
MIT License
18.81k stars 1.04k forks source link

Fix and test cases for unstable tokenization schemes like phi3 #830

Closed Harsha-Nori closed 4 months ago

Harsha-Nori commented 4 months ago

Some models like phi-3 have unstable tokenizers that aren't reversible in select circumstances. This can cause stability issues with guidance's attempt to stay on distribution and retokenize before prompt inputs. I don't believe we have a choice but to skip this step under malformed tokenizers.

This isn't really in a merge-ready state, but is a bandaid fix and some reproducible tests to start a discussion. If we expect this to be a trend, we should formalize the exception process, but if it's a small number of models, the hack in TransformersEngine and Engine classes may be the minimal code change we need.

(Following a discussion with @slundberg and @paulbkoch -- tagging for your awareness)

Harsha-Nori commented 4 months ago

test failures appear unrelated.