snowplow / snowplow-rdb-loader

Stores Snowplow enriched events in Redshift, Snowflake and Databricks
Other
31 stars 17 forks source link

AWS streaming transformer should use default chain to set S3 region #1346

Closed istreeter closed 5 months ago

istreeter commented 7 months ago

In most places snowplow apps let the aws sdk figure out the region using the default chain. But AWS streaming transformer uses hadoop for writing to S3 (only for parquet output format) and infers the region slightly differently.

With this change, we use the AWS SDK to infer the region and then explicitly pass it to hadoop.

This means the transformer should work in other AWS partitions, e.g. aws-china.