search

snowplow-incubator / snowplow-lake-loader

Snowplow Lake Loader
Other
0 stars 3 forks source link

Lake Loader Support For Iceberg #22

Open devorbit opened 1 year ago

devorbit commented 1 year ago

Hi Team,

I have recently started exploring Snowplow for building a client event analytics platform. Currently, our data platform is built on GCP and Apache Kafka. While exploring some of the options and limitations around loading data to a data lake and warehouse, I came across the 'snowplow-lake-loader' option.

As of now, this app/lib only supports the Delta format, but BigQuery doesn't support external tables in Delta yet. I wanted to check softly if we are planning for Iceberg support anytime soon in the future.

Thanks Jay

istreeter commented 1 year ago

Hi Jay,

Iceberg is absolutely an important Open Table Format that we want to support next. Especially as it has good integrations with warehouses like Snowflake and BigQuery. We will announce over on Snowplow's Discourse once we have added this feature.

By the way, the first release of the lake loader already has partial support for Iceberg with BigQuery. We did not announce this feature yet, because the overall experience is a bit rough-around-the-edges still and I want to improve the experience a bit before we claim we support it properly.