LightEval is a lightweight LLM evaluation suite that Hugging Face has been using internally with the recently released LLM data processing library datatrove and LLM training library nanotron.
Theoretically everything up to 1.0.0 is considered unstable and prone to change at any time:
Major version zero (0.y.z) is for initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable.
At Hugging Face, we try and apply the rule that while we have major version zero, minor releases (0.x.0) may add new features and break things (the equivalent of a major release), while patch releases (0.0.x) behave very closely to patches as understood by semantic versioning:
Patch version Z (x.y.Z | x > 0) MUST be incremented if only backward compatible bug fixes are introduced. A bug fix is defined as an internal change that fixes incorrect behavior.
Now for development versions, this is a bit out of the control of semantic versioning IMO but the way that we do it in transformers/diffusers/huggingface_hub is to update the version defined in the init to the "development of the next version".
What that means is that:
For transformers, where there are no plans to go to v5 for now, we would do as such:
We release version v4.38.0
The next version we'll release, if everything goes well, is going to be v4.39.0
We put v4.39.0.dev0 as we're developing that next version.
We're not putting v4.38.1.dev0 as that would mean "we're preparing for the upcoming patch; patch which may or may not happen, and that should only contain bugfixes and no new features.
For huggingface_hub, there are no plan to go to v1 for now, so:
We release version v0.21.0
The next version we'll release, if everything goes well, is going to be v0.22.0
We put v0.22.0.dev0 as we're developing that next version.
If, however, we were planning on releasing a v1 for huggingface_hub, we might instead put v1.0.0.dev0. We won't want to put v0.21.1.dev0 at any point however.
Finally, we only put .dev0 right now (and no .dev1, .dev2, etc), but we could eventually add them with patches as separators. So we'd go:
v4.38.0 Released
v4.39.0.dev0 in the init
v4.38.1 Released
v4.39.0.dev1 in the init
Here what's most important is that you stick to it, imo.
Theoretically everything up to 1.0.0 is considered unstable and prone to change at any time:
At Hugging Face, we try and apply the rule that while we have major version zero, minor releases (
0.x.0) may add new features and break things (the equivalent of a major release), while patch releases (0.0.x) behave very closely to patches as understood by semantic versioning:Now for development versions, this is a bit out of the control of semantic versioning IMO but the way that we do it in
transformers/diffusers/huggingface_hubis to update the version defined in the init to the "development of the next version".What that means is that:
For
transformers, where there are no plans to go to v5 for now, we would do as such:v4.38.0v4.39.0v4.39.0.dev0as we're developing that next version.We're not putting
v4.38.1.dev0as that would mean "we're preparing for the upcoming patch; patch which may or may not happen, and that should only contain bugfixes and no new features.For
huggingface_hub, there are no plan to go to v1 for now, so:v0.21.0v0.22.0v0.22.0.dev0as we're developing that next version.If, however, we were planning on releasing a v1 for
huggingface_hub, we might instead putv1.0.0.dev0. We won't want to putv0.21.1.dev0at any point however.Finally, we only put
.dev0right now (and no.dev1,.dev2, etc), but we could eventually add them with patches as separators. So we'd go:v4.38.0Releasedv4.39.0.dev0in the initv4.38.1Releasedv4.39.0.dev1in the init Here what's most important is that you stick to it, imo._Originally posted by @LysandreJik in https://github.com/huggingface/lighteval/pull/77#discussion_r1514537715_