Evaluate Apache Airflow Operator

[lukegonis]

Describe the issue

• The purpose of this issue is to evaluate the Value/Effort of installing the Apache Airflow Operator

Additional Context

• https://airflow.apache.org/
• Email request from AG: The Attorney general has been looking into an ETL and workflow management engine to enable us to address some of our modernization complexities. We have been running POC with Apache Airflow and we thought it might be nice to have this installed as CRD operators on the Openshift cluster. https://catalog.redhat.com/software/containers/stackable/stackable-apache-airflow-operator/641882500f4e10c3c2d50d52 https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html#using-official-airflow-helm-chart Let me know what we need to do to get started possibly in KLAB. Thank you

Definition of done Four key dimensions to consider:

• [ ] Service Design: gauge community interest
• [ ] Ops: evaluate technical feasibility and impact (i.e. compatibility issues, platform maintenance impact, long term effects, etc..)
• [ ] SRE: evaluate service/operator reliability measures and monitor-ability
• [ ] Cost: evaluate license/deployment cost and operational cost (FTE/person hours)

[sookeke]

Thanks for adding this to your spring @lukegonis

[StevenBarre]

Installed in KLAB via the Operator Hub so Stanley can evaluate it alongside other options for ETL.

[StevenBarre]

@sookeke any updates on generating some community interest in this operator? Any updates on testing the operator in KLAB?

[StevenBarre]

Blocked waiting on feedback from customer

[StevenBarre]

Closing due to lack of feedback

[lukegonis]

Re-opening this issue per request from community member. Next step is to evaluate the effort required to install/maintain.

[lukegonis]

for more info on the request/requestor, see this RC thread: https://chat.developer.gov.bc.ca/channel/devops-how-to?msg=ehnPMvYtGkAiYDNZh

[StevenBarre]

Installed into KLAB and informed customer to test it out.

• Ops: evaluate technical feasibility and impact (i.e. compatibility issues, platform maintenance impact, long term effects, etc..)
  • So far the operator was easy to install and didn't require much extra fiddling.
  • It does create a storage class called "secrets" that might need some changes to the quotas handed out by the provisioner?
  • Will require an extra hour during upgrades to check release notes and verify which version is compatible with the new OpenShift version
• Cost: evaluate license/deployment cost and operational cost (FTE/person hours)
  • Available as Community edition or with a subscription https://stackable.tech/en/plans/#subscription

[StevenBarre]

Blocked waiting on feedback from AG

[StevenBarre]

Installed in KLAB. Waiting feedback from AG on if it works.

[StevenBarre]

I should be able to test it in early July.

[StevenBarre]

User is still testing, will circle back Aug 16th