github-actions[bot]
commented
1 week ago Hi there 👋 Thanks for your contribution!
The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.
Let us know if you need any help!
sonarcloud[bot]
Describe your changes:
Added file format, location path and external table lineage to
GlueSource.AWS Glue connector is quite poor in comparison to what you can find e.g. in AWS console. Some of the interesting features, like lineage, we can find in Athena connector - however, Glue tables can be queried by other engines, such as Trino. Athena is not a popular solution for companies holding huge amounts of data, due to costs. Fetching storage metadata in Trino is difficult, so adding them to Glue instead is a quick win.
Changes summary:
GlueSourcenow inherits fromExternalTableLineageMixin.test_table_namesis fixed - with no patching,get_tables_name_and_typewas throwing warnings and not returning anything, hence the test was iterating over an empty result and not asserting anything.#
Type of change:
#
Checklist:
[x] I have read the CONTRIBUTING document.
[ ] My PR title is
Fixes <issue-number>: <short explanation>[x] I have commented on my code, particularly in hard-to-understand areas.
[ ] For JSON Schema changes: I updated the migration scripts or explained why it is not needed.
[x] I have added tests around the new logic.
[x] For connector/ingestion changes: I updated the documentation. -->