search

turboderp / exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs
MIT License
3.53k stars 272 forks source link

beam search support #396

Closed ovowei closed 3 months ago

ovowei commented 6 months ago

I notice that there are beam search parameters in settings of exllama, but they are missing in exllamav2. Is beam search supported or will be supported in the future?

turboderp commented 3 months ago

Referring this to #84. Stay tuned.