Investigate what's wrong with huggingface integration

Dan-wanna-M / formatron

Formatron empowers everyone to control the format of language models' output with minimal overhead.

MIT License

151 stars 6 forks source link

Investigate what's wrong with huggingface integration #4

Closed Dan-wanna-M closed 2 months ago

Dan-wanna-M commented 2 months ago

Interestingly huggingface integration run 10x slower than vllm and exllamav2 integrations. We need to know why this happens, since huggingface is still the default choice for many researchers and developers.

Dan-wanna-M commented 2 months ago

Fixed in v0.2.0.