Add a new inference model backend based on exllama.
Most of the work on this backend was done by Occam. My main contribution was discovering and working around a bug in torch.multinomial, hooking up stoppers, configuring bad_words_ids, and some other minor bug fixes.
Add a new inference model backend based on exllama. Most of the work on this backend was done by Occam. My main contribution was discovering and working around a bug in
torch.multinomial
, hooking up stoppers, configuringbad_words_ids
, and some other minor bug fixes.