Open steffengr opened 1 week ago
@hsheth2 Thank for looking at this! I don't see related tests in this module. Could you point me at an example? I don't have much experience with this code base.
@steffengr we have an integration test (https://github.com/datahub-project/datahub/blob/master/metadata-ingestion-modules/airflow-plugin/tests/integration/test_plugin.py) that runs a number of DAGs (e.g. https://github.com/datahub-project/datahub/blob/master/metadata-ingestion-modules/airflow-plugin/tests/integration/dags/snowflake_operator.py)
That'd probably be the easiest way to do it. Otherwise, we could also do a more targeted unit test in https://github.com/datahub-project/datahub/blob/master/metadata-ingestion-modules/airflow-plugin/tests/unit/test_airflow.py
The GenericSqlExtractor which is currently by the DataHub Airflow plugin to extract lineage information does not properly support the AthenaOperator and crashes with "AttributeError: 'AthenaOperator' object has no attribute 'sql'". This patch introduces a AthenaOperatorExtractor following the BigQueryInsertJobOperatorExtractor example to fix support for the AthenaOperator.
Fixes #11160
Checklist