set ignore EOS by using TEXT_GENERATION_SERVER_IGNORE_EOS_TOKEN

huggingface / tgi-gaudi

Large Language Model Text Generation Inference on Habana Gaudi

http://hf.co/docs/text-generation-inference

Apache License 2.0

27 stars 47 forks source link

set ignore EOS by using TEXT_GENERATION_SERVER_IGNORE_EOS_TOKEN #234

Closed schoi-habana closed 1 month ago

schoi-habana commented 1 month ago

QA's benchmark scripts set this env var TEXT_GENERATION_SERVER_IGNORE_EOS_TOKEN=true but it's not used any place. This change overwrite ignore_eos_token in the stopping criteria when the env var is true. It will help benchmarks get a fixed number of output tokens.

mandy-li commented 1 month ago

@regisss , pls review and merge if ok to you. thanks