pass volumes/volume-mounts and env-vars through to gitsync containers

[adwk67]

Description

See https://github.com/orgs/stackabletech/discussions/49. Also part of https://github.com/stackabletech/issues/issues/548.

This is to enable SSH-usage using volumes/volumeMounts/envOverrides (which should be available anyway): incorporating this into the CRD is not part of this ticket.

CI tests :green_circle: https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/airflow-operator-it-custom/136/

OKD :green_circle:

--- PASS: kuttl (806.00s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/smoke_airflow-2.8.1_openshift-true_executor-kubernetes (215.72s)
        --- PASS: kuttl/harness/cluster-operation_airflow-latest-2.8.1_openshift-true (269.89s)
        --- PASS: kuttl/harness/logging_airflow-2.8.1_openshift-true (188.91s)
        --- PASS: kuttl/harness/orphaned-resources_airflow-latest-2.8.1_openshift-true (176.65s)
        --- PASS: kuttl/harness/ldap_airflow-latest-2.8.1_ldap-authentication-server-verification-tls_openshift-true_executor-kubernetes (195.68s)
        --- PASS: kuttl/harness/mount-dags-gitsync_airflow-latest-2.8.1_openshift-true_executor-kubernetes (223.35s)
        --- PASS: kuttl/harness/mount-dags-configmap_airflow-latest-2.8.1_openshift-true_executor-kubernetes (144.86s)
        --- PASS: kuttl/harness/resources_airflow-latest-2.8.1_openshift-true (127.48s)

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

# Author
- [x] Changes are OpenShift compatible
- [x] Helm chart can be installed and deployed operator works
- [x] Integration tests passed (adapted gitsync test)
- [x] Changes need to be "offline" compatible

# Reviewer
- [x] Code contains useful comments
- [x] Code contains useful logging statements
- [x] (Integration-)Test cases added (adapted gitsync test)
- [x] Documentation added or updated. Follows the [style guide](https://docs.stackable.tech/home/nightly/contributor/docs-style-guide).
- [x] Changelog updated
- [x] Cargo.toml only contains references to git tags (not specific commits or branches)

# Acceptance
- [ ] Feature Tracker has been updated
- [ ] Proper release label has been added
- [ ] [Roadmap](https://github.com/orgs/stackabletech/projects/25/views/1) has been updated

[PaulienVa]

@adwk67 Thank you lots for merging this so fast! when will this be released?

[adwk67]

You can use the nightly version of the operator chart already (version e.g. airflow-operator:0.0.0-dev). The next release (24.7) is due by the end of next month.

[lfrancke]

Is there anything to document here?

[adwk67]

This was a bugfix as they should have been passed through anyway. Now the code hopefully matches the docs :)

[PaulienVa]

@adwk67 we are testing the nightly atm and it is not working yet.

We are connecting the git-sync container through https, so we need to use some certificates. Again; for the webserver and the scheduler it works perfectly fine. However when starting off a DAG with a kubernetesExecutor the property http.sslCAInfo configured like documented here, using the gitSyncConf in the AirflowCluster Resource does not get propagated through the git-sync container of my kubernetes Executor. Hence my DAGs cannot connect to git and do not run.

Is it possible to fix this in the shortest terms; this still is an inconsistent state of the operator. (webserver and scheduler working fine, executors not)

[razvan]

@adwk67 we are testing the nightly atm and it is not working yet.

Hey, are looking into this.

Would you mind in the meantime, opening a new issue and providing the exact definition of the AirflowCluster object ?

Thank you.

[PaulienVa]

Will do! Thank you!