search

astronomer / astronomer-cosmos

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
https://astronomer.github.io/astronomer-cosmos/
Apache License 2.0
749 stars 170 forks source link

Add support for openlineage when running from Kubernetes/Docker #496

Closed tatiana closed 9 months ago

tatiana commented 1 year ago

Follow up #305 and #485

Openlineage support and dataset URIs only work for the LOCAL and VIRTUALENV execution modes ATM. In order to support DOCKER and KUBERNETES, we'll have to expose the Manifest/Run results via XCom, since openlineage-integration-common depends on them.

dosubot[bot] commented 9 months ago

Hi, @tatiana,

I'm helping the Cosmos team manage their backlog and am marking this issue as stale. The issue you raised seeks to enable support for openlineage when running from Kubernetes/Docker. Currently, openlineage support and dataset URIs are only functional for the LOCAL and VIRTUALENV execution modes. To extend this support to DOCKER and KUBERNETES, the Manifest/Run results must be exposed via XCom, as the openlineage-integration-common relies on them. There hasn't been any further activity or comments on the issue.

Could you please confirm if this issue is still relevant to the latest version of the Cosmos repository? If it is, please let the Cosmos team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days. Thank you!