search

huggingface / lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
MIT License
689 stars 78 forks source link

Homogeneize logging system #150

Closed clefourrier closed 2 months ago

clefourrier commented 5 months ago

Make it so both model types can run tensorboard evals.

This should be good to go, but will need to wait for an update in huggingface_hub before merging, as the lib stops the job if ŧensorboard is installed when it's not needed (see Slack for discussions), and cc @Wauplin :)

Closes #118

Wauplin commented 5 months ago

Opened https://github.com/huggingface/huggingface_hub/pull/2205 for the huggingface_hub part :)

clefourrier commented 5 months ago

Thanks :) In which huggingface_hub release will this be included?

Wauplin commented 5 months ago

Thanks :) In which huggingface_hub release will this be included?

0.23.x

Not sure when it'll be though (not before ~10 days I think, given last release was recent and not much has been done since then).

Wauplin commented 5 months ago

A possible solution if you want a quick fix is to install from source. Not the best if the lib' is used "in prod".

clefourrier commented 5 months ago

We'll wait for your release then! :) (It's not used in prod but we want to keep the pytoml clean-ish)

Wauplin commented 5 months ago

Understandable!

clefourrier commented 5 months ago

@Wauplin is your release coming soon? :)

Wauplin commented 5 months ago

Not at short term actually (with ~10 days). I haven't pushed much changes to huggingface_hub since this fix. You would prefer something more short-term I guess?

clefourrier commented 5 months ago

If it were possible to push a patch for this it would be great, but else we'll leave it hanging - it's a part of the code we're not usually touching a lot so it won't be a hassle to merge in the future.

Wauplin commented 5 months ago

I'd prefer not to do a patch release for that as other "import-related" stuff has been fixed as well in huggingface_hub and cherry-picking commits would be hazardous (and taking from main would not be a patch release).

Another solution is to rely on the main branch for 2 weeks and I can take care of pinning again the version once huggingface_hub 0.23 is out.

clefourrier commented 5 months ago

Nope we'll wait - it's better for us to only rely on releases to avoid unexpected silent failures - but thanks :)

NathanHB commented 4 months ago

hey @Wauplin ! any news on the state of the release ? :)

Wauplin commented 4 months ago

Sorry sorry, forgot to let you know before going on holidays :see_no_evil: Huggingface_hub 0.23 has been shipped 3 weeks ago :grimacing: https://github.com/huggingface/huggingface_hub/releases/tag/v0.23.0

clefourrier commented 4 months ago

@NathanHB tested it and it should be good and mergeable when you've got the time :)