Closed tensiondriven closed 11 months ago
exllama_hf is not complete. Some things don't work, like contrastive search or perplexity evaluation. Possibly no_repeat_ngram is one of those things
no_repeat_ngrams works; num_beams does not.
(I conflated these because I was using both in my testing.)
Ergonomics-wise, if these issues are known, it would make a lot of sense to me to be defensive and explicitly throw when an unsupported feature is used, since it will result in an exception anyway
does exllama (non-hf) work with num_beams?
[edit] it doesn't seem to crash like exllama_hf but I don't think it's working. VRAM usage isn't spiking and I'm not seeing it alter its own text as it's being generated like I normally do
just wanted to mention for those with VRAM to spare running with bnb 4bit is a viable alternative for num_beams
This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.
Describe the bug
While attempting to work with exllama_hf, I discovered that passing num_beams > 1 during inference when using exllama_hf results in an exception (see below).
Additional notes:
--loader exllama
--loader exllama_hf
--loader exllama_hf
--loader exllama
This was discovered while diagnosing https://github.com/oobabooga/text-generation-webui/issues/3028
Is there an existing issue for this?
Reproduction
1) Run text-generation-webui:
2) Perform inference via API or UI with
num_beams: 2
(or any value greater than 1)Screenshot
No response
Logs
Note: the value
[2, 162]
seems to depend on the context length.