Possibility to read apiKey for custom repositories in Iglu.json from an environment variable?

[neonfink]

Is your feature request related to a problem? Please describe. For our local Snowplow Micro testing setup we are reading the schemas from a custom repository, which we added to list of repositories inside our iglu.json like this:

{
    "name": "Development Repo Name",
    "priority": 1,
    "vendorPrefixes": [
      "com.snowplowanalytics.etc"
    ],
    "connection": {
      "http": {
        "uri": "https://custom-repository-host-name",
        "apikey": "ACTUAL APIKEY"
      }
    }
}

This approach works fine locally, however, if we want to include Snowplow Micro validations in our CI pipeline, hardcoding API key in the config file is a no-go.

Describe the solution you'd like Obtaining the apikey value from an environment variable seems like the most obvious and straightforward approach. Having went through Snowplow Micro getting started guide and Testing Tutorial I did not find anything mentioning how to handle this particular configuration caveat. Perhaps it's already possible but simply is not documented? 👀

Describe alternatives you've considered We could amend the iglu.json on the fly as part of the build step and insert the apikey value in a hacky way but that feels like an unnecessary workaround, maybe I'm missing some obvious details, any suggestions are welcome. Thanks in advance!

[adrianoesch]

I just worked my way around this issue by (1) using a custom image and (2) using a custom entrypoint script as suggested above. Not very nice, but better than nothing:

(1) Dockerfile

FROM snowplow/snowplow-micro:latest

# snowplow default user doesn't have install permissions
USER root 

# install sed to replace text in file
RUN apt-get install sed 

# copy config files to /tmp so you can rewrite them within docker (mounting a volume would change file on docker host too)
COPY . /tmp/. 

COPY entrypoint.sh .
ENTRYPOINT ["./entrypoint.sh"]

(2) entrypoint.sh

#!/bin/sh
echo "IGLU_SERVER_KEY:" $IGLU_SERVER_KEY
sed -i 's/${IGLU_SERVER_KEY}/'"$IGLU_SERVER_KEY"'/' /tmp/iglu.json
/opt/snowplow/bin/snowplow-micro --collector-config /tmp/micro.conf --iglu /tmp/iglu.json

[benjben]

We should do like in Enrich and read the resolver as a hocon, to benefit from environment variable substitution.

[stanch]

If you just want to add a single registry with no vendor prefixes, this should work for you: https://docs.snowplow.io/docs/getting-started-with-micro/basic-usage/#pointing-micro-to-an-iglu-registry (just added in Micro 1.5.0). We are still planning to enable environment variables in advanced configuration (iglu.json) though.