search

EleutherAI / lm-evaluation-harness

A framework for few-shot evaluation of language models.
https://www.eleuther.ai
MIT License
6.34k stars 1.68k forks source link

Empty --log_samples outputs #2115

Open IsraelAbebe opened 1 month ago

IsraelAbebe commented 1 month ago

I checked similar issues

While working with this tool for machine translation generate_until task all calculation give correct output but after doing --log_samples in wandb i see the following where most of the precitions give null response. i am using llama-3-8B model. any idea what the problem is?

Screenshot 2024-07-19 at 08 01 18
haileyschoelkopf commented 1 month ago

Hi! Could you share the command you're running and what the locally logged files look like?

Could you also share the codebase version you are using to run these commands? (e.g. PyPI version number, or a git commit from the main branch)