PecanProject / pecan

The Predictive Ecosystem Analyzer (PEcAn) is an integrated ecological bioinformatics toolbox.
www.pecanproject.org
Other
199 stars 231 forks source link

[Docker, CI] Allow building pecan/depends from pecan/depends #3260

Closed infotroph closed 1 month ago

infotroph commented 4 months ago

Description

For discussion -- definitely needs input from @robkooper before merging!

Adds a third option for the dependency image when building Docker images: In addition to "use the existing image" and "build from scratch", adds "try to update an existing image".

Also edits the Docker CI job to check for changed dependencies when building Docker images in a PR branch, and if detected to update the depends image before building other images on top of it.

Additionally and unrelatedly, adds workflow_dispatch hooks to several more workflows for easier debugging.

Motivation and Context

3237 is currently failing the Docker CI check because it introduces new version requirements on dplyr and dbplyr that will be satisfied at Depends build time but that Make doesn't know how to resolve. Trying to fix this in the Makefile would add a lot of only-rarely-used complexity, and pre-emptively rebuilding the depends image from scratch on every CI run would be slow and wasteful.

Running the depends build only when it will actually change anything (i.e. when the PR touches docker/depends/*) is a first step, but it still feels wasteful to wait for all hojillion deps to reinstall when we really only added one or two. Then I realized that the two time-consuming steps in the depends build (apt updates and R package installation) will both skip packages that are already present on the disk -- so if we start from an existing pecan/depends image, adding a layer with the changed dependencies should be fast.

Review Time Estimate

Types of changes

Checklist:

infotroph commented 4 months ago

@robkooper Discussion questions:

infotroph commented 3 months ago

/style