Determine if Airflow can be made independent of Open Metadata

os-climate / os_c_data_commons

Repository for Data Commons platform architecture overview, as well as developer and user documentation

Apache License 2.0

21 stars 10 forks source link

Determine if Airflow can be made independent of Open Metadata #243

Open HeatherAck opened 1 year ago

HeatherAck commented 1 year ago

Currently open metadata and airflow are tightly coupled. Determine if airflow can be externalized to enable other functionality.

caldeirav commented 1 year ago

Deployment as of now has a stand-alone airflow. To be tested.

caldeirav commented 1 year ago

Right now I am unable to to create a new connection from OpenMetadata in the Data Mesh environment as it is not able to access the Airflow Managed APIs

Added https://github.com/opendatahub-io-contrib/data-mesh-pattern/issues/74 for tracking

caldeirav commented 1 year ago

We need to steup airflow to external OpenMetaData as per:

https://docs.open-metadata.org/v1.1.x/deployment/ingestion/openmetadata

This means we need to install openmetadata-ingestion package with the connector plug -in and openmetadata-managed-apis module package to Airflow image.