search

elementary-data / elementary

The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
https://www.elementary-data.com/
Apache License 2.0
1.93k stars 165 forks source link

Allow file format change for Glue adapter #1711

Open Jeremynadal33 opened 1 month ago

Jeremynadal33 commented 1 month ago

Is your feature request related to a problem? Please describe. This is to complete issue #177 . I added PRs dbt-data-reliability and elementary for allowing Glue adapter with Delta file format. It then requires a configuration that allow Delta tables from the Glue session. We should add the possibility to use other file formats.

Describe the solution you'd like I think we should add a macro that allow a new configuration to be passed to choose the file format with a verification that the file format allows incremental loads.

Describe alternatives you've considered N/A

Additional context Not sure if that is relevant or possible but I feel like we should check if the Glue session is able to use the file format.

Would you be willing to contribute this feature? I would like to but I don't have time for now...

haritamar commented 1 month ago

Hi @Jeremynadal33 ! Thanks a lot for your contribution of the Glue adapter, and for this issue. Will look at the contribution when I can, in any case makes sense this is a separate issue.