search

huggingface / lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
MIT License
689 stars 78 forks source link

[FT] Remove obsolete config properties (frozen, output_regex) #305

Open hynky1999 opened 2 weeks ago

hynky1999 commented 2 weeks ago

Request

Removed obselete config properties:

For more context see: https://github.com/huggingface/lighteval/pull/289

the output_regex is legacy from the harness, we needed it for bigbench tasks but this is now handled by the metrics direclty. Frozen was to make sure the task did not change over time but we are now using task versioning for this.
clefourrier commented 2 weeks ago

Agree on both. Note: frozen was initially for tasks that we had thoroughly debugged and that were included in the test suite, hence that were not expected to change definition as they were robust. So it's also an indicator of higher quality.