stackabletech / airflow-operator

Stackable Operator for Apache Airflow
Other
21 stars 2 forks source link

Allow arbitrary python code #493

Closed Maleware closed 2 weeks ago

Maleware commented 2 weeks ago

Description

This PR is the outcome of the decision https://github.com/stackabletech/decisions/issues/21

Contains:

Definition of Done Checklist

# Author
- [ ] Changes are OpenShift compatible
- [ ] CRD changes approved
- [ ] CRD documentation for all fields, following the [style guide](https://docs.stackable.tech/home/nightly/contributor/docs/style-guide).
- [ ] Helm chart can be installed and deployed operator works
- [ ] Integration tests passed (for non trivial changes)
- [ ] Changes need to be "offline" compatible
# Reviewer
- [ ] Code contains useful comments
- [ ] Code contains useful logging statements
- [ ] (Integration-)Test cases added
- [ ] Documentation added or updated. Follows the [style guide](https://docs.stackable.tech/home/nightly/contributor/docs/style-guide).
- [ ] Changelog updated
- [ ] Cargo.toml only contains references to git tags (not specific commits or branches)
# Acceptance
- [ ] Feature Tracker has been updated
- [ ] Proper release label has been added
- [ ] [Roadmap](https://github.com/orgs/stackabletech/projects/25/views/1) has been updated
Maleware commented 2 weeks ago

Tests green: 🟢

--- PASS: kuttl (206.16s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/smoke_airflow-2.9.2_openshift-false_executor-celery (206.12s)
PASS
lfrancke commented 1 week ago

The decision is not public, can you summarize what this is about and if it is something that is documented please include a link to the docs. If it's not documented can you briefly summarize why note?

Does this need to be mentioned in the release notes?

Maleware commented 1 week ago

This is to allow the user to add arbitrary python code to the webserver_config.py.

We've got it documented in our superset docs and airflow docs

However, this can be used for any case where superset allows us to replace functions with custom ones. It was driven by superset documentation about custom OIDC manager and we found that we should add it to airflow too.

We've reached a decision on that this is how airflow wants to be configured ( no matter how ugly it might appear ) and thus we want the operator to be useful in this case.

We've had an customer for airflow who wrote their whole config for themself ( basically ignoring the operators work and thus it's not used ) to achieve having own classes and functions. They now can use the configs the operator provides and have their custom code as a config.

I can't really make a objective statement weather or not we should mention it in the release notes. For me it should as it enables some customers and they might wanna know.

Left to do: Do we want to add python libraries to the Airflow Dockerfile to have OIDC supported properly, if a customer chooses to do so?

lfrancke commented 1 day ago

Thank you. In that case: Can you please add a short sentence as it could appear in the release notes as a comment here?