search

huggingface / lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
MIT License
779 stars 90 forks source link

`--no_multichoice_continuations_start_space` should also cover startof word token #45

Open clefourrier opened 8 months ago

clefourrier commented 8 months ago

If the tokenizer prepends _ as sow token, it will make single token evals fail. Reported by @anton-l

clefourrier commented 3 months ago

@anton-l do you remember in which case you encountered this?