search

huggingface / evaluate

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.
https://huggingface.co/docs/evaluate
Apache License 2.0
2.02k stars 259 forks source link

Add note in repo and docs that library is no longer actively maintained #612

Closed MoritzLaurer closed 1 month ago

MoritzLaurer commented 2 months ago

AFAIK the library is no longer actively maintained. Users are still building upon the library and it might be better for them to use maintained libraries like light-eval. Adding a small note about this in the gh readme and the hf docs page could help avoid confusion. See this internal discussion on Slack for context. @lvwerra

Kami-chanw commented 2 months ago

Can someone explain why HF gives up this library?

MoritzLaurer commented 2 months ago

double checking with @lhoestq @lvwerra, cc @clefourrier @lewtun (see this internal thread for context)

lewtun commented 2 months ago

cc also @LysandreJik for viz

lhoestq commented 2 months ago

afaik there is still maintenance happening for major cases, but not for adding new features or small bug fixes atm

nehalecky commented 1 month ago

Hi HF team! We're also interested to know what your plans are for maintaining this repo. We look forward to hearing back from you—thanks 🙌

LysandreJik commented 1 month ago

Thanks all for your comments! We're indeed slowly moving away from evaluate as the default evaluation repo for us at Hugging Face as we have not found it to be sufficiently useful for the community.

As Moritz mentions, we are maintaining the lighteval repository for evaluation and are also significantly using the LLM Harness from EleutherAI in our stack. We recommend using those for the latest evaluation developments.

These two aren't complete replacements for what evaluate aims to be, so we're still monitoring this repo and merging important fixes. We're driven by community adoption and usage, so if your work is hindered by the slowed-down pace of development on this library, please react to this message or comment below, as knowing so will help us put the right amount of effort into it.