Run release in CI

[wslulciuc]

We closed #826 to moving towards a multi-module project structure. We've introduced release.sh, but some improvements can still be made to unify the release.

Most of the current release steps are run locally. They are:

1. Bump and release marquez-python
2. Bump marquez-python dependency in marquez-airflow
3. Bump and release marquez-airflow
4. Bump development versions for the java modules
5. Tag main branch with X.Y.Z
6. Publish docker images (api, web)
7. Publish java modules to maven

The proposed release steps below would run in CI to release Marquez X.Y.Z. First, we would tag main branch with X.Y.Z to kicks off the CI release workflow, then:

1. Publish java modules to maven
2. Publish docker images (api, web)
3. Bump and release python modules

[mobuchowski]

@wslulciuc created draft PR for this: https://github.com/MarquezProject/marquez/pull/1168

What I've done:

1. Python: split test and build phases for marquez-python and marquez-airflow. Build phases are triggered only on tag (recognized by env variable set by CI - $CIRCLE_TAG. Build phases are ment to create Python wheels and put them to shared workspace. Then, release step - also, only triggered by tag - one for all python projects, just pushes wheels created in shared workspace to pypi.
2. Docker: least changes - basically all I had to do is script bumping version in gradle-properties and add dependency on build-web and build-api steps.
3. Maven: after bit of tinkering on my own I've found that gradle release plugin has option -Prelease.releaseVersion=version which does all the bumping for me. Found that we can sign binaries on CI using env variables. Still, couldn't reliably test if last step works - which is publishing to oss.sonatype.org with publishing plugin. By looking at docs I don't see how it could differ, provided that we have all the env variables set up, but that would be on you to check :wink:

[wslulciuc]

@mobuchowski: I've looked over your draft PR, great work! I have some suggestions:

1. Python: split test and build phases for marquez-python and marquez-airflow. Build phases are triggered only on tag (recognized by env variable set by CI - $CIRCLE_TAG. Build phases are ment to create Python wheels and put them to shared workspace. Then, release step - also, only triggered by tag - one for all python projects, just pushes wheels created in shared workspace to pypi.

I really like the idea of having a build phase (separate from testing) that builds the wheels for each python module and saves the distribution to a shared workspace. This allows the release-python CI job to easily handle other python modules we may add with no extra effort!

I looked over / tested the release script for python (bump-python-version.sh) in your PR. I think we'll want to checkin the release bump into git before we build the wheels. The idea is that when we tag a commit, the version that we plan to release has already been committed and is staged for an official release. That is, say I checked out the tag X.Y.Z, the python module versions should also be X.Y.Z. That way we know for sure what was part of the release. I recently opened #1193 that cleans up our python versioning and should simplify things.

2. Docker: least changes - basically all I had to do is script bumping version in gradle-properties and add dependency on build-web and build-api steps.

Only comment here (similar to the python modules) is that we'll want to modify the gradle-properties file before we tag a commit for a release.

3. Maven: after bit of tinkering on my own I've found that gradle release plugin has option -Prelease.releaseVersion=version which does all the bumping for me. Found that we can sign binaries on CI using env variables. Still, couldn't reliably test if last step works - which is publishing to oss.sonatype.org with publishing plugin. By looking at docs I don't see how it could differ, provided that we have all the env variables set up, but that would be on you to check

Yep, happy to tests the publishing to sonatype once your PR is merged! That said, I think this step should only have the publish command, while, as mentioned above, modifying the gradle-properties file will happen before we tag a commit for a release :)

I think we only need to make a few minor changes to your PR, and it's mainly on when certain files are modified. Here's what I would suggest as release steps:

1. Bump the version in the root gradle.properties and the setup.py for each python module
2. Tag the release with X.Y.Z
3. CI kickoffs the release jobs
4. Prepare the next development version

So, steps 2, 3, and 4 are pretty straightforward as it doesn't change much in your proposal. But, in step 1, we can easily use your regex to modify the version in the root gradle.properties (note, we no longer need to also have the version specified in the gradle.properties files for each module, just the root). For each python modules, we can use:

$ bump2version manual --new-version NEW_VERSION

Note that since the part arg is required for bump2version, I went with manual, but it can be named anything (see bump2version#131.

What are your thoughts on having a script called new-version.sh <new-version> that does the following?

• Sets the version in gradle.properties to <new-version>
• Sets the version for marquez-python to <new-version>
• Sets the version for marquez-airflow to <new-version>

Then we tag a commit and continue on with 3, 4

[mobuchowski]

Agreed, that does make sense, and bumping version before tagging, as separate manual step does fix the issues you're talking about. I'll change the PR.

new-version.sh <new-version> :ok_hand:

[mobuchowski]

Added this script to this PR new-version.sh. Now we can either merge this, and change bumpversion usage in your PR, or merge your PR first and change it here.

[wslulciuc]

Fixed #1168