retrofit PDS python repos to use new versioning scheme with roundup

[jordanpadams]

Is your feature request related to a problem? Please describe. New versioneer scheme has been implemented per NASA-PDS/pds-template-repo-python#2 . Now we need to figure out how we version and tag a python repo.

some questions we need to consider:

• how do we want to trigger a versioning and build with roundup? "release" branch?
• OR do we trigger a versioning and build with roundup, or is this something that should be apart of pds-github-util?

this Epic also includes the various updates that need to be made to roundup for triggering and publishing releases

this comment from https://github.com/NASA-PDS/pdsen-corral/issues/16#issuecomment-749106266 may also be useful:

@tloubrieu-jpl @nutjob4life I think the current python versioning issue and the newly created issues on roundup-action are all related to some rethinking of how we should handle releases with github actions.

I think there are a few questions / issues we need to address to make sure the actual stable release is stable :-) .

1. when / how do we trigger the github action? a. Current state: triggered by a git tag. this means that version update and git tag are manual steps that need to be performed on the repo. b. Proposed improvement: triggered by some other "pre-release" step, e.g. release branch or version branch (e.g. v0.0.0 or some specific commit message to trigger. github action handles versioning, tagging, etc. we would need this trigger to somehow contain the updated version so the software knows what to increment the version to (major, minor, patch)

2. missing Github releases (NASA-PDS/roundup-action#25) a. Current state: roundup only pushes releases for unstable, but skips that step for stable. b. Proposed improvement: after tagging, github release should be created

3. missing site to github pages (NASA-PDS/roundup-action#28) a. Current state: roundup packages up the documentation and pushes to the release (although this doesn't work right now since a release is not created for stable releases) b. Proposed improvement: roundup packages up the docs, but also needs to push those docs to github pages branch (note update for maven project to use site:stage goal to group all docs together)

4. bug updating version after tag was created (NASA-PDS/roundup-action#26) a. *Current state: roundup (or pds-github-util) increments version after the software has already been tagged, but that version should be updated prior to tagging b. Proposed improvement:** use the version in the tagged release as-is

thoughts?

[nutjob4life]

⑴⒝ sounds like the better way to go due to the automation but I recommend avoiding a specifically formatted commit message as that suffers from the in band signaling problem. I like the idea of having a release branch to which we can trace all releases, and it would make triggering actions trivial.

⑵⒝ agreed.

⑶⒝ sounds reasonable.

⑷⒝ Maybe it is better to just use the version tag as set by a human instead of having an automated process bump the number? For micro releases of of the format v.major.minor.micro maybe it's not so bad but automation can't be the judge of what's a minor or a major release. Having a human set the tag explicitly in the process helps us be mindful of what goes into each release, perhaps?

I like the idea of "release early, release often" but a little extra "sanity check" could help?

[jordanpadams]

⑴⒝ sounds like the better way to go due to the automation but I recommend avoiding a specifically formatted commit message as that suffers from the in band signaling problem. I like the idea of having a release branch to which we can trace all releases, and it would make triggering actions trivial.

+1

⑷⒝ Maybe it is better to just use the version tag as set by a human instead of having an automated process bump the number? For micro releases of of the format v.major.minor.micro maybe it's not so bad but automation can't be the judge of what's a minor or a major release. Having a human set the tag explicitly in the process helps us be mindful of what goes into each release, perhaps?

+1 . Maybe once the action tags and pushes the release it can increment the version to the next -dev / -SNAPSHOT?

[tloubrieu-jpl]

On 1)b), compared to the branching model there https://nvie.com/posts/a-successful-git-branching-model/ Would our release branch be equivalent to master there ? And our master equivalent to their dev branch ?

I know we moved away at some point from this complex branching model but if we are going back that way (which I am ok with) we should use the same conventions as https://nvie.com/posts/a-successful-git-branching-model/ or some other reliable guidelines.

[jordanpadams]

@tloubrieu-jpl @nutjob4life i am ok with going back to that mechanism and setting everyone's default branch to develop if we want to go there. honestly, thinking about adding hot fixes, this would actually really simplify things as well. any time you merge to master, it triggers the release step.

the only complication is having everyone port over to using that as the base branch. looks like Microsoft uses a release branch idea and has main always be the latest: https://docs.microsoft.com/en-us/azure/devops/repos/git/git-branching-guidance?view=azure-devops

[nutjob4life]

Hi folks, good discussion. Some thoughts:

The nice thing about git is that it supports so many different workflows. The unfortunate aspect about git is that it supports so many workflows 😉. Finding a balance between ease of use, flexibility, and proneness to errors can be tricky. You've already identified a few potential ideas. Here are a few more, with some brief highlight of their characteristics:

• "Centralized". A terrible workflow, suitable for really small teams and people who like resolving merge conflicts.
• "Feature Branch". A simple workflow that takes advantage of one level of branches (one per feature, bug fix, etc.) and centers around pull requests for dialog and sign-off of them. This seems closest to what PDS EN is doing right now.
• "Github Flow". This is a highly agile workflow that Github itself uses to deliver Github, i.e., for the day-to-day running of github.com. I think I read once that Github delivers updates to github.com daily, so clearly there's some merit to this workflow. See Github's illustrated guide for more information, but especially this essay on why they prefer it to Gitflow.
• "Gitflow". A complex workflow with develop and main (née master) branches, plus branches for features, hotfixes and support. This is the workflow @tloubrieu-jpl mentioned above. The fact this is implemented with a third-party add-on tool may be a red flag for the complexity of this workflow, but you can do it without the add-on as well. See Vincent Driessen's essay and Atlassian's overview for more information.
• "Forking Workflow". Not appropriate as we do in fact use a central repository, i.e., github.com, and everyone has access.

One issue that I had with Gitflow has been identified by Vincent Driessen in his updated introduction on the essay: that it's more appropriate for delivered software. For software that is provided as service and is never made available as a product to download, the Github Flow works better. The Planetary Data System seems to be slowly moving in this direction, but giving that we're still having to tell people how to download and install Python, we're not there yet. That suggests Gitflow, with the proviso that @jordanpadams recognized: we have to train people off of master and onto develop.

(But that's not so bad; it's also an opportunity to get rid of negative connotations of master and move to main; see renaming.)

However, Gitflow is yet another mental layer for all of us to have to deal with. We're not a huge team and we don't have a lot of developers overlapping work on a single large repository so far as I am aware. Perhaps sticking with what we're currently doing (Feature Branch workflow) is appropriate?

(I am also out of time for today, so I'd also like to ask that we move issue #32 to the Marion Jones milestone.)

[jordanpadams]

@nutjob4life @tloubrieu-jpl I am open to staying the same or Gitflow. I guess this goes back to the original question at hand then. what will the release and deploy github action key off of?

If we want a release branch to key off of, this is eerily similar to Gitflow, just have a separate release branch off main.

Or we can take the Microsoft release approach, using some release/xxx (or some other naming scheme) (e.g. release/1.1.0, and our builds key off of that?

[nutjob4life]

I'm leaning towards the Azure team's approach because there's some nice semantic bonus along with it. The action could look for a push to release/[0-9]\.[0-9]\.[0-9]. Thoughts?

[jordanpadams]

so workflow:

1. create and push release/0.0.0 branch
2. roundup action pulls version number from branch name (or throws error if invalid), updates repo version, then some ordering of tags repo, deploy, release, deploy docs
3. merge release branch to main
4. update main with new SNAPSHOT / dev version

thoughts?

[tloubrieu-jpl]

Yes that make sense to me and I like the fact that we are relying on someone else strategy for branch/release (azure).

Regarding the release branch name, I was thinking to create the branch without the patch version number, for example release/1.2

And for the actual release we can have tags with the patch version (1.2.0, then 1.2.1...).

Since the patch number is meant for hot fix on the release.

[jordanpadams]

@tloubrieu-jpl so I think you are saying if we use release/1.2, roundup would kick-off any time the branch is created OR any update occurs to that branch:

• go out to github and check if any 1.2.* releases exist
• if not, update the version to 1.2.0, tag the current branch, release, etc..
• if yes, increment the patch version (e.g. 1.2.1), update the version to 1.2.1, tag the current branch, release, etc..., and release/1.2 now has the latest hotfix

[nutjob4life]

So a partial automation solution? A human creates & pushes release/X.Y and the system does X.Y.0, X.Y.1, X.Y.2, by itself. We are in charge of major.minor but the system takes care of .patch for us.

I think I can deal with that. The .patch levels are tedious but we get to exercise our judgment when it comes to the rest.

(This approach is orthogonal to the actual workflow, which probably is a separate issue: do we want to make "Gitflow" the standard for NASA PDS or keep the feature branch approach.)

[nutjob4life]

Hey @tloubrieu-jpl @jordanpadams , I just want to confirm if you think this approach is correct. The current GitHubReleaseStep (for Python packages) does essentially this:

if isStable(): return  # Do nothing
token = getToken()
exec('git fetch --prune --unshallow --tags')
for tag in exec('git --list *dev*'):
    exec('git tag --delete {tag}')
    exec('git push --delete origin {tag}')
exec('python-snapshot-release --token {token}')

Now we'll want it to do this:

token = getToken()
exec('git fetch --prune --unshallow --tags')
for tag in exec('git --list *dev*'):
    exec('git tag --delete {tag}')
    exec('git push --delete origin {tag}')
if isStable():
    tag = getPushedTag()
    exec('git tag -a -m {tag} release/{tag}')  # note below
    exec('git push origin {tag}')
    exec('python-release --token {token}')
else:
    exec('python-snapshot-release --token {token}')

where getPushedTag() returns:

• 1.2.3 if the push was release/1.2.3
• 1.2.3 if the push was release/1.2 and a previous release with tag 1.2.2 exists
• 1.2.0 if the push was release/1.2 and no previous release with tag 1.2.* exists

which I think we can get with a shortcut via git describe --tags --abbrev=0.

Regarding the note below, I think this step can fail if the user pushes an explicit fully-described major.minor.micro tag, i.e., release/1.2.3 and so it re-tags release/1.2.3 so it should softly ignore that.

Also question: is there any difference between python-release and python-snapshot-release? Glancing at the code it looks like they invoke the same module+function. Unless they look at argv[0] and behave differently?

(Also I'm typing all this up during a conference so I'm waaaay distracted 🤪)

[jordanpadams]

@nutjob4life looks right to me. per python-release vs. python-snapshot-release there is not difference. i simple kept python-snapshot-release for backwards compatibility. the tool used to only be used for snapshots, but then we updated it to do more than that.

[tloubrieu-jpl]

Thanks @nutjob4life, I feel like this piece of code:

for tag in exec('git --list dev'): exec('git tag --delete {tag}') exec('git push --delete origin {tag}')

should be under the else statement.

[tloubrieu-jpl]

Or actually we don't want to have dev tags anymore (never) ? If yes, the code should be ok.

[nutjob4life]

I guess maybe it's okay to always delete dev tags? 🤷‍♀️

[jordanpadams]

@tloubrieu-jpl @nutjob4life is that true? We definitely still want snapshots for java repos. Is there a reason we are not doing that for python repos?

Also, did we determine the user will version the actual app prior to pushing the branch? Or should this step also be running versioneer / mvn set version?

[nutjob4life]

In Java terms a "snapshot" release is a release in the central repository (Sonatype's OSSRH) with the flag "-SNAPSHOT" in the artifact ID. In Python terms it's … uhm … is it a release on test.pypi.org?

Also, I thought the user would do a push to releases/X.Y and the automation would figure out correct value of Z for X.Y.Z, right?

[jordanpadams]

@nutjob4life i am ok with releases/X.Y. also, per -dev, I think that makes sense. Having a -dev release doesn't really help anything. as long as we have consistent instructions for building the latest in-dev code, I am OK with that (maybe we update the Python template with some consistent dev build instructions?)

[tloubrieu-jpl]

@nutjob4life @jordanpadams For the python snapshot, we could have X.Y-{commit hash} as produced by versionneer, for example sds-core-0.3.0-5-ge3f036d.tar.gz

Then I am not sure how we can document this dependency with pip and requirements.txt. I am having 0+unknown as the version matcher for one project. I read that we can use --pre option to get requirement on unstable release. I am not sure how this can work.

[jordanpadams]

@tloubrieu-jpl for all of that, maybe that is too much down in the weeds? if they want the dev version, maybe we just direct them to install from source with pip and call it good?