Standalone nanotron config

huggingface / lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

MIT License

851 stars 100 forks source link

Closed hynky1999 closed 2 months ago

hynky1999 commented 3 months ago

This PR moves the lighteval config to lighteval codebase.

Enforces the lighteval_config_path as the only way to read the lighteval config. The nanotron part is ignore, this way the breaking changes won't be as breaking.
Some typing corrections Comments
Haven't tested yet as I don't want to rebase on nanotron fix PR so waiting for that to be merged

NathanHB commented 2 months ago

I took a look, tested and fixed a few bumps, it should work tell me what you thikn !

hynky1999 commented 2 months ago

@NathanHB Thanks for fixing the problems. I made small nits to the config (outpid_dir type to str, as that's what it's in cli interface).

Two more things:

I changed the nanotron model to take batch_size from override_bs (as do accelerate models), I still think it's very confusing, but leaving it as it is
I noticed that batch detection for nanotron doesn't work. Created an issue for that. I think it's low prio but let's keep track of it https://github.com/huggingface/lighteval/issues/286