Stop conditions and exclude prompt for Base generator

turboderp / exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs

MIT License

3.28k stars 243 forks source link

Stop conditions and exclude prompt for Base generator #224

Open SinanAkkoyun opened 7 months ago

SinanAkkoyun commented 7 months ago

I needed the base gen to work with custom stop conditions (mainly for batched inference reasons). Now, stop_conditions (str list) and multiple stop_tokens (compatible with supplying an ID instead of ID list) are implemented

I also added an optional param to exclude the prompts from the output