upload assets on tag - Githubissues

ome / openmicroscopy

OME (Open Microscopy Environment) develops open-source software and data format standards for the storage and manipulation of biological light microscopy data. A joint project between universities, research establishments and industry in Europe and the USA, OME has over 20 active researchers with strong links to the microscopy community. Funded by private and public research grants, OME has been a major force on the international microscopy stage since 2000.

https://www.openmicroscopy.org/omero

GNU General Public License v2.0

200 stars 102 forks source link

upload assets on tag #6318

Closed jburel closed 2 years ago

jburel commented 2 years ago

Follow up from 5.6.4 release. This PR adds steps to the build workflow to avoid the manual migration of the artifacts from downloads.o.org to GitHub.

The workflow does the following, When a tag is pushed:

The artifacts are built
A release is created
The artifacts are uploaded

I have opted for one single SHASUMS and MD5 collecting the various values to simplify the selection and reduce the number of uploaded files.

This can replaced the "release" job on CI

See https://github.com/jburel/openmicroscopy/releases/tag/v5.6.4-m1

sbesson commented 2 years ago

Immediate thoughts:

using the GitHub release API as the primary source of artifacts feels like a valuable move following the same line of argument as https://github.com/ome/omero-documentation/pull/2195#issuecomment-1113022648,
comparing to https://downloads.openmicroscopy.org/omero/5.6.4/artifacts/, the staging release linked above above omits openmicroscopy-5.6.4.zip. Or is it equivalent the source code zip?
both the build number and the ice separator are removed from the bundle names. I think the former makes sense as it was related to the Jenkins release infrastructure. The latter was meaningful when several Ice versions were concurrently supported so as long as we accept this will not be the case moving forward, it makes sense as a simplification.
With the name simplification, the proposal defines a fairly simple URL for server artifacts https://github.com/ome/openmicroscopy/releases/download/v<version>/OMERO.server-<version>.zip
- is the expectation that all downstream consumers (omero-install, omero-documentation, omego, www.openmicroscopy.org) will be updated to start consuming this new URL? or would we retain some mirroring logic and leave some dependencies on downloads.openmicroscopy.org/omero/?
- this also raises the timeliness of reviewing the repository name as mentioned in https://github.com/ome/omero-documentation/pull/2195#issuecomment-983522960

jburel commented 2 years ago

openmicroscopy-vXX.zip is equivalent to the source.zip. Note that it is not available on GitHub on the tag 5.6.3/5.6.4
downstream repositories will consume the zip from GitHub (as it was for omero-install prior to 5.6.4). I don't think there is too much to gain to mirror to downloads.o.org
When we have doc on RTD, changing the link and updating the doc will be straightforward but there are not really coupled. Changing to the various repositories could be done ahead of time. This could potentially be done via an action so we keep repositories in synch i.e. new openmicroscopy tag, PR opened in another repo similar to what we do for the doc and omeroweb-install and omero-install

joshmoore commented 2 years ago

One thing to keep in mind that I went through with https://github.com/zarr-developers/zarr-logo/pull/2/commits is that on download things get automatically zipped.

Also: I can highly recommend experimenting with the gh release commands to make sure what's being upload and downloaded matches your expectations.

sbesson commented 2 years ago

Another component which I forgot to mention in my previous comment is our Ansible deployment stack especially ome.omero_server and the underlying omego library currently rely on the current https://downloads.openmicroscopy.org/omero layout and naming for retrieving the server artifacts. Amongst other, this proposal might force us to revisit https://github.com/ome/omego/pull/127 to download components directly from GitHub.

Something to think about is the breaking impact on deployment if a new library/role release or new scripts are required to upgrade a server. I don't consider it to be a blocker per se but it might lead us to a dedicated upgrade documentation page similar to what we did for the Python3 migration.

jburel commented 2 years ago

Reactivate that work

Use a new workflow
Add ice36 to the name
use gh release command
The zip do not get "zipped" twice See https://github.com/jburel/openmicroscopy/releases/tag/v5.6.5-m1

jburel commented 2 years ago

Add openmicroscopy-XXX.zipsee https://github.com/jburel/openmicroscopy/releases/tag/v5.6.5-m2

jburel commented 2 years ago

cf. https://github.com/ome/openmicroscopy/pull/6318#issuecomment-1113450540 Re-activated see https://github.com/ome/omego/pull/132.

Adding support for latest should prevent the upgrade of link in https://github.com/ome/omero-install/blob/develop/linux/step04_all_omero_install.sh#L15

jburel commented 2 years ago

A few inline comments and a couple of questions:

the new process seems to fully drop the build suffix. Is that on purpose and do we know whether this might cause impact downstream?

I can't get a build number as bXXXX is internal to our CI. The build is now happening in GHA.

sbesson commented 2 years ago

I can't get a build number as bXXXX is internal to our CI. The build is now happening in GHA.

I assume in the GitHub context github.run_id would be the closest equivalent. As mentioned above, I do not know of the impact of dropping the build. it An advantage is that would certainly make the download URL fully predictable from the OMERO.server version.

jburel commented 2 years ago

Using the job id is an option

jburel commented 2 years ago

https://github.com/jburel/openmicroscopy/releases/tag/v5.6.5-m3 using the run_id in the name This should match the convention currently used when the artifacts are built via Jenkins CI

jburel commented 2 years ago

Looking closely, I don't think the build number needs to be kept. I am also looking into improving https://github.com/ome/omego/pull/132 so we can pass major or major.minor so we can install the latest version from a major version without having to update the link

the usage of get_latest_runs will need to be reviewed. This is mainly (only!) used to retrieve a latest run from Jenkins latest.

jburel commented 2 years ago

Looking at another part, candidate for automation, not including the build number will allow to create the url by finding the latest release tag from the GitHub API . Including the build number will not permit that.

jburel commented 2 years ago

Build number is used in https://github.com/ome/www.openmicroscopy.org/blob/master/omero/downloads/index.html. In that case it already points to GitHub
Ansible role do not rely on build number
Build number in omego is used for downloading from latest CI. The number used in this PR will never match leading to potential confusion i.e. 2 different build numbers

jburel commented 2 years ago

For the build number, it will be matter of adjusting a few places. If we agree, I will roll back and do not add the build number A rebuild will be happening by pushing a new tag. One downside of this approach is if we want to retract a release. For that eventually e.g. major releases, rc will be used or possible intermediate dev tags.

jburel commented 2 years ago

Last 2 commits

Remove the build number
simplify the code. All actions occur in the target directory so the name in SHASUM does not content target/NAME

Tested via https://github.com/jburel/openmicroscopy/releases/tag/5.6.5-m5

jburel commented 2 years ago

Any more comments on this PR?

sbesson commented 2 years ago

Is the missing leading 5 in https://github.com/jburel/openmicroscopy/releases/tag/5.6.5-m5 on purpose?

jburel commented 2 years ago

oh. Let me check. That should not be the case.

Edit: This is because I did not tag with vX.X.X and I remove the first character i.e. tag_value="${tag_name:1}-ice36" I will add check if it starts with v to avoid such mistake

jburel commented 2 years ago

The last commit has been tested using 2 tags. One starting with v and one without v https://github.com/jburel/openmicroscopy/releases/tag/5.6.5-m7 https://github.com/jburel/openmicroscopy/releases/tag/v5.6.5-m6