search

huggingface / lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
MIT License
690 stars 78 forks source link

Relax lower bound on `transformers` dependency? #72

Closed lewtun closed 7 months ago

lewtun commented 7 months ago

Currently lighteval sets the min required version of transformers to 4.38.0. Is this strictly needed for some features in the lib?

If not, I suggest rolling back to a slightly older version so that lighteval can be integrated into other libs that user older versions of transformers (otherwise one gets an incompatibility error).

In general I would be a bit careful about bumping the lower bound over time as it will tend to break any integrations that pin a specific version of transformers

PS for our use case, it's not a real issue to bump transformers!

clefourrier commented 7 months ago

It's for compatibility with gemma models etc. Which lib would you need that for?

lewtun commented 7 months ago

It's for compatibility with gemma models etc. Which lib would you need that for?

Ah no worries! We use v37.2 in the H4 codebase, but easy to bump it :)