This PR consolidate the recent efforts towards refine.bio Docker images build process optimization.
Every image name is now based on the branch hash value and cache layers are uploaded into a registry for further sharing purpose. This allowed retiring Docker.affymetrix_local image and related prepare_image.sh logic.
The API image has been split into base and local/production images.
The affymetrix/agilent tests are 2 separate jobs now and use larger runners (-m suffix).
These changes resulted in >50% (~1 hour vs ~30 minutes) tests execution time improvement: run a or run b VS this PR run.
Some points to keep in mind during review:
Name Docker images based on the branch hash value (shasum)
Add setup.py PEP-440 workaround for common package
Reorganize GHA jobs
Utilize Docker registry cache for image cache layers sharing
Use separate Docker images (_cache) for caching
Optimize Docker images size (foreman, salmon)
Add remote builder support (used by default in GHA jobs)
Set umask explicitly to avoid cache miss due to permissions diff
Introduce API base image, derive api local/production from it
Reformat affected files for better readability
Use better names for scripts/variables
Consolidate renv related scripts location (use common/ everywhere)
Retire -d option of prepare_image.sh
Delete affymetrix_local image as no longer used (affymetrix cache boost satisfies our tests execution time requirements)
Sort .dockerignore entries
Update .pre-commit-config.yaml
Update README
pre-commit-terraform has been disabled temporarily due to unrelated failures.
The PR contains all changes required for further staging/production deploy process optimization identified at this moment.
Checklist
[x] Lint and unit tests pass locally with my changes
[ ] I have added tests that prove my fix is effective or that my feature works
[x] I have added necessary documentation (if appropriate)
[ ] Any dependent changes have been merged and published in downstream modules
Issue Number
Closes #3288
Purpose/Implementation Notes
This PR consolidate the recent efforts towards refine.bio Docker images build process optimization.
Every image name is now based on the branch hash value and cache layers are uploaded into a registry for further sharing purpose. This allowed retiring
Docker.affymetrix_localimage and relatedprepare_image.shlogic.The API image has been split into base and local/production images. The affymetrix/agilent tests are 2 separate jobs now and use larger runners (-m suffix).
These changes resulted in >50% (~1 hour vs ~30 minutes) tests execution time improvement: run a or run b VS this PR run.
Some points to keep in mind during review:
prepare_image.shpre-commit-terraformhas been disabled temporarily due to unrelated failures.The PR contains all changes required for further staging/production deploy process optimization identified at this moment.
Checklist