NOAA-GSL / ExascaleWorkflowSandbox

Other
2 stars 2 forks source link

Remove broken caching and use workaround for broken pip install #28

Closed christopherwharrop-noaa closed 1 year ago

christopherwharrop-noaa commented 1 year ago

This PR does four things:

  1. Removes use of docker layer caching from CI workflow. The docker layer caching doesn't work properly because there isn't enough memory on the runner to hold both the cache and the built images. Even when the cache is read, docker-compose runs out of disk space when starting the containers. Docker compose V2 was tried, but it is also broken and fails to find the image it just created (see this). Attempts to upgrade docker compose did not solve the problem.
  2. Works around Spack/pip conflicts. Spack and pip do not play well together. This is a known issue. During the flux-core and flux-sched installations, also gets installed as a dependency. However, it mysteriously disappears when the install of flux completes. This reinstalls pip, which simply updates the Spack view without rebuilding it, so that Parsl may be installed with pip. However, another problem arises because when parsl is installed via pip, pip does something strange that causes a cffi package incompatibility which breaks flux. So after installing Parsl via pip, flux must be reinstalled to reinstate the correct cffi package back into the Spack view. Once all this is done, a functioning Parsl and Flux are installed.
  3. Move the container registry to ghcr.io and use Github packages.
  4. Add a CI badge the README.
christopherwharrop-noaa commented 1 year ago

@NaureenBharwaniNOAA - This is finally ready. There were a LOT of problems that I had to debug and it took a long time to sort through all the various issues.