search

astronomy-commons / axs

Astronomy eXtensions for Spark: Fast, Scalable, Analytics of Billion+ row catalogs
https://axs.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
23 stars 12 forks source link

GitHub Actions to Build and Release AXS Distributions #18

Closed stevenstetzler closed 3 years ago

stevenstetzler commented 4 years ago

Starting a PR to track issue of automating releases of AXS.

See .github/workflows/release.yaml and .github/workflows/build.yaml for github actions scripts.

stevenstetzler commented 4 years ago

@ctslater I think I'm almost ready to merge these changes to get automated builds of AXS going. I wanted to get your input first though on how we deal with versioning of axs vs axs-spark.

The workflow for this build is: 1) Push commit with a tag v*, e.g. v1.0 2) axs version is extracted from the tag, e.g. v1.0. The script will do three things with that: a) check out the commit that tag references, b) build Spark with the command ./dev/make-distribution.sh --name AXS-$AXS_VERSION where AXS_VERSION is pulled from the tag, and c) name the official GitHub release "Release {{ tag }}" 3) We check out a specific tag/branch reference of the astronomy-commons/axs-spark repository to build against. I am fixing this to axs-3.0.0-preview in the build script. This is set with the ref key for the "Checkout AXS-Spark" build step in the script .github/workflows/release.yaml 4) If the build succeeds, it pushes a release to GitHub with an attached tar ball named axs-distribution.tgz. The release name will be "Release {{ tag }}", e.g. "Release v1.0"

For example, the latest release at https://github.com/stevenstetzler/axs/releases is built from a commit with tag v3.0.0-preview

Question 1: do we want to build AXS against different versions of AXS-Spark? For example, we could create several of these scripts that build against tags axs-2.3.0, axs-2.4.3, or axs-master in addition to axs-3.0.0-preview. We could also just change this hardcoded value in release.yaml for different commits of AXS so that a tag in axs will map to just one tag of axs-spark instead of building against all of them.

Question 2: do we have any tests we need to run to verify that AXS works? Or are we just relying on make-distribution.sh not failing?

ctslater commented 4 years ago

I think that overall sounds like the right plan. One modification: can we use a stand-alone text file to define the tag (or maybe a sha?) of axs-spark that we want to incorporate in a given build? That might give the linkage between the two repos a bit more visibility (I worry a little bit about things in dot directories being somewhat hidden.) I think that also implicitly answers your question #1: any given commit of axs should only be built against a specific version of axs-spark.

One other thing is that we could either incorporate the axs-spark repo by downloading a binary distribution from that repo or building it during the axs build. Either should accomplish the same thing, so I don't have a strong preference.

Question 2: I've been manually checking releases before giving them out. I wouldn't block the automated build effort on also getting better automated tests, we'll get to those eventually. There are lots of unit tests inside axs-spark.

stevenstetzler commented 3 years ago

I've changed the build script to just build the AXS jar and make a draft release that we can edit before making an official release. Builds with trigger when tags are pushed in the format "v*", e.g. "v1.0".

I've also squashed my various commits into one.

stevenstetzler commented 3 years ago

Also, AXS includes python code...should we consider creating a PyPi package for it?

ctslater commented 3 years ago

I'm going to merge this into a branch on astronomy-commons/axs so I can try it out more easily.