search

snowplow / snowplow-rdb-loader

Stores Snowplow enriched events in Redshift, Snowflake and Databricks
Other
31 stars 17 forks source link

Environment variable resolution in HOCON config #1227

Closed piankris closed 1 year ago

piankris commented 1 year ago

I saw that this feature is implemented in the collector, enricher and loader but not here. Is there a reason why the rdb module can't be configured the same way?

istreeter commented 1 year ago

Hi @piankris thanks for pointing out it doesn't work -- I wasn't aware of this. I'll look into it, and hopefully get a fix into one of the next releases.

istreeter commented 1 year ago

Hi @piankris we recently released RDB loader 5.5.0, and we improved how the hocon parsing works. It would be great if you can give it a try and let us know if it works for you.

You can now pass a the configuration files as arguments to the loader like this:

docker run \
  -v /path/to/config:/myconfig \
  snowplow/rdb-loader-redshift:5.5.0 \
  --config /myconfig/loader.hocon \
  --iglu-config /myconfig/resolver.json

...and your config hocon file and iglu resolver file can contain environment variables, e.g. ${API_KEY}

istreeter commented 1 year ago

I'm going to close this, because as far as I know this was fixed by #1252 . Feel free to open if there is still something not working in the latest versions.