snowplow / snowplow-rdb-loader

Stores Snowplow enriched events in Redshift, Snowflake and Databricks
Other
31 stars 17 forks source link

Loader should not call the "list" Iglu endpoint for Snowflake/Databricks tables #1350

Closed istreeter closed 4 months ago

istreeter commented 4 months ago

Jira ref: PDP-1274

When loading to Redshift, the loader needs to call the "list" Iglu api endpoint in order to discover all minor versions of a schema. It uses the list to merge the schemas.

But for Snowflake/Databricks tables the loader does not need to merge schemas, so there is no need to call the "list" endpoint.

In RDB Loader version 6.0.0 we accidentally changed the loader so it calls the "list" endpoint regardless of the table type. This is a problem for Snowflake/Databricks loads because: