search

huggingface / lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
MIT License
687 stars 78 forks source link

Expose `stop_sequence` at command line #196

Open lewtun opened 4 months ago

lewtun commented 4 months ago

Models like llama-3 use a chat template where the expected stop sequence is <|eot_id|> instead of the common EOS token used in other models. This means that generative benchmarks like ifeval continue generating past the EOS token and give incorrect results.

One way to handle this would be to either include this special token as a default in the generative benchmarks or alternatively expose --stop_sequence as an argument in the main script that users can control.

clefourrier commented 2 months ago

Hi @lewtun , just to clarify, is <|eot_id|> a different stop token for the chat template specifically? And is the model using <|end_of_text|> for the rest?

Other question, do you think fixing #16 would be enough to solve this too?