Run TokenEnforcer flow in all unit tests

noamgat / lm-format-enforcer

Enforce the output format (JSON Schema, Regex etc) of a language model

MIT License

1.01k stars 46 forks source link

Run TokenEnforcer flow in all unit tests #45

Closed noamgat closed 6 months ago

noamgat commented 6 months ago

This PR makes the testing flow nearer to real world usage, including a Llama2 tokenizer with real-world challenging cases. It also allows the unit tests to serve as a better performance / profiling benchmark to improve performance.