search

Dan-wanna-M / formatron

Formatron empowers everyone to control the format of language models' output with minimal overhead.
MIT License
151 stars 6 forks source link

Check opportunistic masking #6

Closed Dan-wanna-M closed 3 weeks ago

Dan-wanna-M commented 2 months ago

It is possible to sample a token first and check whether that token can be accepted by the formatter. This means we only need to compute the mask for the whole vocabulary if the token is not accepted. This nonetheless does not integrate well with existing pipelines. We probably need to get some PRs passed first.

Dan-wanna-M commented 3 weeks ago

We are already fast even without it.