Support for Iceberg Tables in Snowflake with AWS Glue Catalog Integration

dbt-labs / dbt-external-tables

dbt macros to stage external sources

https://hub.getdbt.com/dbt-labs/dbt_external_tables/latest/

Apache License 2.0

314 stars 123 forks source link

Support for Iceberg Tables in Snowflake with AWS Glue Catalog Integration #251

Open amirbtb opened 9 months ago

amirbtb commented 9 months ago

Describe the feature

Support for Iceberg tables in Snowflake with AWS Glue catalog integration.

Describe alternatives you've considered

Currently, there is no existing alternative that would allow us to directly reference Iceberg tables (managed by catalog integration with AWS Glue catalog) as a dbt (external) source. We would need to create tables in Snowflake that reference the Iceberg tables and then we could use them as dbt sources.

Additional context

Database-specific: The feature is specific to Snowflake. It could be extended to BigQuery as I heard it also supports Iceberg via BigLake but I'm not very familiar with the latter.
Context: Our organization uses AWS Glue catalog for cataloging and S3 for storage.

Who will this benefit?

This feature would be useful for anyone :

Using Iceberg with Snowflake
Using AWS Glue catalog integration

amirbtb commented 9 months ago

I found this comment, but I'm not sure if this feature is supported and if that works with AWS Glue catalog integration :

Note, the way that https://github.com/dbt-labs/dbt-external-tables/pull/138 is implemented, option 1 via "external tables" should already work.


- name: iceberg_tbl
description: "External table using Iceberg files"
external:
location: "@stage"                      # reference an existing external stage
file_format: "( type = parquet )"       # fully specified here, or reference an existing file format
table_format: iceberg                   # specify the table format
auto_refresh: false                     # requires configuring an event notification from Amazon S3

github-actions[bot] commented 1 month ago

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

dataders commented 1 month ago

@amirbtb do you still have this use case? Have you seen the new pattern of EXTERNAL TABLE + ICEBERG TABLE compared to this packages use of EXTERNAL STAGE + EXTERNAL TABLE?