search

turboderp / exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs
MIT License
3.53k stars 274 forks source link

Grammar Support on Exllamav2 #284

Closed tednas closed 3 months ago

tednas commented 8 months ago

Using Exllamav2 inference script (not chatbot interface), do we have any alternative for the capability of using grammar in llamacpp?

tednas commented 8 months ago

Any idea @turboderp ?

c608345 commented 4 months ago

Grammar is supported in tabbyAPI. https://github.com/theroyallab/tabbyAPI/pull/71

turboderp commented 3 months ago

Indeed. There's also an example script for LMFE.