[CT-1492] [Feature] Add support for Iceberg Tables

[teej]

Is this your first time submitting a feature request?

• [X] I have read the expectations for open source contributors
• [X] I have searched the existing issues, and I could not find an existing issue for this feature
• [X] I am requesting a straightforward extension of existing dbt-snowflake functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Snowflake has announced support for Iceberg Tables, an external table format but with full interoperability as a 1st class snowflake table.

Once this feature goes live, dbt-snowflake should support materializing a model as an external iceberg table.

Describe alternatives you've considered

No response

Who will this benefit?

• dbt users who want their models available as parquet files in storage they control
• dbt users who want an easy way to interoperate with non-dbt systems that use the iceberg table format
• dbt users who want direct access and control over the underlying storage of their data

Are you interested in contributing this feature?

yes

Anything else?

Iceberg Tables are in private preview as of 2022-11-10

[erniedeferia]

+1

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[jaklan]

Any chance to put more light on that? We really need a Snowflake equivalent of https://github.com/dbt-athena/dbt-athena to push forward the adoption of open table format (which, I believe, would be also pretty crucial for dbt to support in the long term).

[multiprocessingh]

+1 here! This would be super useful to support for enterprise sas use cases!

[WangCHEN9]

+1 here!

[DLewisEndeavor]

+1 here! Looking into Iceberg for enterprise sharing across platforms

[apd-bscally]

+1, Iceberg is now in public preview on Snowflake

[mourtallah]

+1 here!

[VSchmidts]

+1 here!

[ksdaly]

+1

[ZergRocks]

+1

[sfc-gh-ripu]

+1

[jimvekemans]

+1, will work on this a bit in the coming weeks and, if successful, open a pull request

[dennysrega1ado]

+1

[waitsfornone]

+1 here. We are adopting Iceberg tables as our data lake format and I would love to generate the tables out of Snowflake with dbt.

[mikezusev]

+1

[zewang-sanofi]

+1

[sfc-gh-wriley]

+1 :-)

[njrogers]

+1

[Fleid]

Hey all! Thanks for "upvoting", please continue to do so. We are wrapping up all the internals work to support 1.8, plus a few more things to do for quality of the plugins and how we test them, and then Iceberg will be at the top of the list!

[sfc-gh-ftrejo]

+1

[sfc-gh-elethier]

+1

[sfc-gh-amdeshpande]

+1

[sfc-gh-mpowell]

+1

[joatkh]

+1

[Jemsley0]

+1

[ncapeta]

Any ETA on 1.8 ?

[jonask137]

Any ETA on 1.8 ?

According to the documentation, it should be released as of April 2024: https://docs.getdbt.com/docs/dbt-versions/core#latest-releases

[shayms8]

i'm getting errors when trying to query a MAP iceberg object from dbt:

  002040 (42601): SQL compilation error:
  Unsupported data type 'map(VARCHAR(16777216), VARCHAR(16777216))'.
  compiled Code at target/run/dbt_snowflake_1/models/L1_iceberg/L1__client_events_iceberg.sql

any idea why it happens? can i somehow solve it? didn't find a documentation suggesting MAP columns are not supported on dbt_snowflake integration

[ShahNewazKhan]

+1

[sfc-gh-wriley]

Is this now LIVE...? Was slated for APRIL release?

[sivakanthavel-tigeranalytics]

+1

[antonio-yuen-zocdoc]

Any news on this?

[dweaver33]

Any update? We're looking into iceberg tables on Snowflake now as well.

[freemanrickhuang]

+1

[colin-rogers-dbt]

We will be prioritizing this work over the next couple of weeks. Please stay tuned.

[davegot12]

+1

[mnoumanshahzad]

+1

[VersusFacit]

Just to update, I'm actually drafting out features right now. We have a table prototype going if you want a sneak peek. Fair warning, we have some performance issues to navigate given Snowflake metadata constraints.

We are actively scoping incremental tables out :)

1170

[TomRutten]

+1

[harveymarshall]

+1

[VersusFacit]

For those following closely along, we just merged the PR I linked above into main. 🚀

its_happening.gif

So, the Iceberg table materialization is live for those who like to live at the bleeding edge of our adapters. (we don't recommend this 😅 )

It will take the next 2-3 weeks for us to QA it for Cloud / release in a formal OSS version.

Caveat

Because of some metadata query latency, this feature will be gated behind a behavior flag: enable_iceberg_materializations. Early adopters will need to use this flag in their project configurations to enable Iceberg features.

Snowflake in a future release will alter their metadata queries to remove this latency hit, at which point we'll remove the need for the flag in the first place.

---

Onto incremental models next.