search

ACCESS-NRI / model-builder

Model build environment and meta-build tooling
0 stars 0 forks source link

Release strategy for build containers #7

Open aidanheerdegen opened 1 year ago

aidanheerdegen commented 1 year ago

Want to have docker containers we can use for CI builds (c.f. #2).

Making these containers available to the ACCESS community would be very beneficial, e.g. @micaeljtoliveira has expressed an interest is using these for COSIMA CI.

In order for these containers to be of use to the wider community there would need to be some way for spack packages to be requested to be added, and also a release strategy formalised so that tagged releases could be used with confidence about what they contain.

Initial possible release strategy:

  1. Tag releases with year month and release number (YYYY.MM.N) (see e.g. https://github.com/dask/dask/tags)
  2. Six monthly update cycle (suggest March/October to avoid holiday dead spots)
  3. Release number starts at 0, e.g. 2023.03.0. Subsequent releases increment release number by 1, e.g. 2023.03.1
  4. Release numbers are required to allow for new packages to be added mid-cycle as required (but existing package versions would remain the same)
  5. Have a dev tagged container where the latest versions of all components are included. This provides a test bed for new compilers and libraries that will be included in the next tagged release

The container file definitions, or the means to generate them, can be stored in a repository. Requests for new packages could be made as a pull request to this repository. If anyone knows of an existing system like this being deployed please reply in a comment with description and a link to relevant repositories.

One possible barrier is Intel OneAPI licensing. Would we be allowed to distribute Intel OneAPI compilers in publicly visible docker container images on the GitHub container repository?

I note e4s does this, e.g. this container

https://hub.docker.com/layers/ecpe4s/e4s-oneapi/22.11/images/sha256-a3be067a792a948ddd4e45be7f29abb665eb61b2b9df2fcc68d73d74bd6954ea?context=explore

RUN /bin/sh -c wget https://registrationcenter-download.intel.com/akdlm/irc_nas/18673/l_BaseKit_p_2022.2.0.262_offline.sh  && bash ./l_BaseKit_p_2022.2.0.262_offline.sh -a --silent --eula accept --components intel.oneapi.lin.ippcp.devel:intel.oneapi.lin.tbb.devel:intel.oneapi.lin.dpcpp_dbg:intel.oneapi.lin.dpcpp-ct:intel.oneapi.lin.dpcpp-cpp-compiler:intel.oneapi.lin.dal.devel:intel.oneapi.lin.dnnl:intel.oneapi.lin.mkl.devel:intel.oneapi.lin.ipp.devel:intel.oneapi.lin.ccl.devel:intel.oneapi.lin.vpl:intel.oneapi.lin.dpl  && rm -f l_BaseKit_p_2022.2.0.262_offline.sh # buildkit

Will lodge an issue with E4S asking about this.

Edit: Added suggestion for specific timing of update cycle

micaeljtoliveira commented 1 year ago

Thanks for the nice summary!

I'm a big fan of semantic versioning, but I understand that it might not be trivial to use here, as many packages included in the containers do not follow it, making it hard to determine backward compatibility.

Also, it would be good to have all the changes to the containers properly documented in a standardized, easy to read changelog (see https://keepachangelog.com/en/1.0.0/).

aidanheerdegen commented 1 year ago

Intel EULA:

https://www.intel.com/content/dam/develop/public/us/en/documents/intel-end-user-license-agreement-for-developer-tools-oct-2020.pdf

Relevant sections:

B. You may utilize a Cloud Provider to host the Materials for You, provided: (i) the Cloud Provider may only host the Materials for Your exclusive use and may not use the Materials for any other purpose whatsoever, including the restriction set forth in Section 4.1(xii); (ii) the Cloud Provider’s use of the Materials must be solely on behalf of and in support of Your Product, and (iii) You will indemnify, hold harmless, and defend Intel and its suppliers from and against any claims or lawsuits, including attorney's fees, that arise or result from Your Cloud Provider’s use, misuse or disclosure of the Materials.

aidanheerdegen commented 1 year ago

Because I have nowhere else to put it, I can't help feeling the tools described in this paper:

https://arxiv.org/abs/2212.07376

and this twitter thread

https://twitter.com/vsoch/status/1603221550232006656

will be useful, but I don't yet know how exactly

https://github.com/singularityhub/guts

https://github.com/singularityhub/shpc-registry-cache