Publish dev releases to PyPi and automatize version computation

[benoit74]

Disclaimer: this is an exploratory issue, I'm still not totally convinced this is the proper way of working, so far this is more a dump of which issue I have and what I've found which could help.

Context

Issue https://github.com/openzim/zimit/issues/300 shows that it might be interesting to publish dev versions to Pypi for proper integration of warc2zim package inside zimit. This would also help to test dev versions more easily for external contributors. Note that by default, pip considers only stable versions, so this is not a problem ; should we want dev versions, we need to use the --pre flag.

Currently, we setup the version manually in main package __init__.py : https://github.com/openzim/_python-bootstrap/blob/3856ce249adb15b4cf37bb7be8bd80e569f1c6e2/src/great_project/__init__.py#L3

This however means that we need an automated way to bump dev versions for every commit on main branch. It would be good to not have to make a new commit just to bump the version, since this would double the number of commits for almost nothing, making the git history quite dirty.

At the same time, we currently rely on manual setup of the next release, meaning we have to manually make commits (again on main branch) at every release to push the proper version, and after release to prepare for next version. Since we also place a tag for the release, there is a significant chance of mismanipulation leading to discrepancy between the tag and the version in Pypi.

We use a major.minor.patch scheme for production, or major.minor.patch-devN for development.

Currently, right after a release we bump the version to the next minor and zero the patch. I.e. if version released is 1.2.3, we bump the version to 1.3.0-dev0. This is not quite appropriate, since we usually never know yet if the next code going to be merged to main is going to be for a patch, for a minor or for a major. Solving this would be great as well.

Proposition

• publish dev versions to Pypi
• remove the version from __init__.py and compute it automatically with versioningit : https://versioningit.readthedocs.io/en/stable/index.html
  • basically we use tags to compute the released version
  • for dev, the version is computed based on last tag + number of commits since this tag
• do not commit current version anymore to source code, this information will be stored in git history
• retrieve version at runtime from importlib.metadata.version("mypackage")

This means we need (but it is a good opportunity) to change our way of working:

• by default, we assumes that next version is going to be a patch of latest released tag (released tag meaning a tag without a .devN suffix)
• once we know we have significant changes needing a minor, the associated merge commit is tagged with the minor dev version (e.g. v1.3.0.dev0).
  • every subsequent commit is automatically numbered based on the distance to latest tag commit, hence being the dev version of next minor
• should we decide to make breaking changes needing a major, again the merge commit is tagged with the major dev version (e.g. v2.0.0.dev0).
  • again, every subsequent commit is automatically numbered based on the distance to current commit, hence being the dev version of next major

Ideally the merge commit going to move from a patch to a minor or major release should be pushed first as a tag, then as a commit to main branch. This way when the CI trigger on push to main branch, the tag is already appropriately incremented (otherwise it will build a 1.2.x.devN instead of a 1.3.0.dev0 for instance). Since we know that this is not going to always be the case (people love to click the "Merge PR" in Github UI), we can modify the publish-dev workflow to also be triggered on push to tags matching the v*.dev0 pattern.

This means we need a custom next_version computation (don't know why it is not standard in versioningit) which:

• compute the next patch if latest tag has no -dev suffix
  • e.g. if latest tag is 1.2.0 (because it has been released), then next commit on main branch will compute next_version as 1.2.1, which once formatted will become 1.2.1.dev1
• keep the same released version if it has a -dev suffix
  • e.g. if latest tag is 1.3.0.dev0 (because we decided to bump to a minor), then next commit on main branch will compute next_version as 1.3.0, which once formatted will become 1.3.0.dev1

I propose to share this next_version method in hatch-openzim package since it is also used to distribute shared stuff across openZIM organization.

Final configuration:

[tool.versioningit.next-version]
method = { module = "hatch-openzim.versioningit", value = "next_version" }

[tool.versioningit.format]
distance = "{next_version}.dev{distance}+{vcs}{rev}"
# Example formatted version: 1.2.4.dev42+ge174a1f

dirty = "{base_version}+d{build_date:%Y%m%d}"
# Example formatted version: 1.2.3+d20230922

distance-dirty = "{next_version}.dev{distance}+{vcs}{rev}.d{build_date:%Y%m%d}"
# Example formatted version: 1.2.4.dev42+ge174a1f.d20230922

WDYT?

[rgaudin]

I'm sorry but I don't understand what problem we are trying to fix with this

Issue https://github.com/openzim/zimit/issues/300 shows that it might be interesting to publish dev versions to Pypi for proper integration of warc2zim package inside zimit.

It's debatable ; that's not what I get from this discussion.

This would also help to test dev versions more easily for external contributors.

I disagree. External contributors who can't setup the dev environment? One of the main goal of this bootstrap is to ease the setup. We are talking about git clone xx ; cd xx ; pip install . vs pip install xxx==x.y.z.devA