search

c0sogi / llama-api

An OpenAI-like LLaMA inference API
MIT License
111 stars 9 forks source link

exllamav2 #20

Open ehartford opened 10 months ago

ehartford commented 10 months ago

Please add support for exllamav2

atisharma commented 10 months ago

https://github.com/c0sogi/llama-api/issues/15

atisharma commented 10 months ago

It seems to be broken at the moment because llama-api is not passing the tokenizer argument to the sampler. ExLlamaV2Sampler.sample() missing 1 required positional argument: 'tokenizer'