This PR introduces the production CDSI Airflow deployment files.
Updates that were made:
Production deployment files stored in digits-eks/eks-prod/shared-services/airflow directory.
Example file for dev ingress stored in digits-eks/eks-dev/shared-services/ingress directory.
Removal of unneeded deployment files. Note that airflow-utils script was removed (doesn't need to be checked into prod repo but can continue to be used locally).
webserverSecretKey value now stored in a k8s secret.
Removed dependency on custom server image - packages that were being installed here were only needed for the NCI DAG and were already being installed in the NCI pod image. The base Airflow server image is now used in override-values.yaml and the custom Dockerfile was removed.
README.md describes deployment steps needed to spin up an Airflow server.
Future cards:
[x] Pin Airflow version for Docker images to avoid unintentional version upgrades. This PR pins apache-airflow:airflow/2.9.2. Pod template Docker image for NCI updated to match: https://github.com/cBioPortal/nci-crdc-pipeline/pull/212
[ ] Move project-specific secrets, variables/connections, and volume mounts to pod template files to simplify override-values.yaml and keep it specific to the server configuration.
[x] Determine if DAG persistence should be enabled. The response to the Airflow PR linked here cautions against use of DAG persistence and gitSync together. - Discussed with Avery 8/15 - going to turn off DAG persistence for Production deployment.
[ ] NCI DAG import error:
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "/opt/airflow/dags/repo/nci-crdc-pipeline/study_builder.py", line 13, in <module>
import jsonpickle
ModuleNotFoundError: No module named 'jsonpickle'
[x] Move pod template file mount to pod template file(s) - Discussed with Avery 8/15 - going to remove the pod template file mount for Production deployment, if needed should be added back in later.
This PR introduces the production CDSI Airflow deployment files.
Updates that were made:
digits-eks/eks-prod/shared-services/airflowdirectory.digits-eks/eks-dev/shared-services/ingressdirectory.airflow-utilsscript was removed (doesn't need to be checked into prod repo but can continue to be used locally).override-values.yaml, now stored inportal-configurationrepo: https://github.mskcc.org/knowledgesystems/portal-configuration/pull/19webserverSecretKeyvalue now stored in a k8s secret.override-values.yamland the custom Dockerfile was removed.README.mddescribes deployment steps needed to spin up an Airflow server.Future cards:
apache-airflow:airflow/2.9.2. Pod template Docker image for NCI updated to match: https://github.com/cBioPortal/nci-crdc-pipeline/pull/212override-values.yamland keep it specific to the server configuration.