Feature/decoding optimization

noamgat / lm-format-enforcer

Enforce the output format (JSON Schema, Regex etc) of a language model

MIT License

1.01k stars 46 forks source link

Closed noamgat closed 6 months ago

noamgat commented 6 months ago

Introduced TokenEnforcerTokenizerData to reuse calculations between token enforcer instances
Improved long sequence performance by only decoding the tokens of the current word being generated