[Feature Request] Leverage Env Vars in YML Files

[rsaltrelli]

The YML files are great for checking configuration into source control, but some information (like the Schema Registry URLs and bootstrap servers, for example) I'd rather keep out of source control and set with environment variables similar to Docker Compose.

default.yml

runners:
    default: "local"
    config:
        lambda:
            deploy: null
            credentials:
                type: "default"
            awsRegion: null
            enabled: false
        kubernetes:
            namespace: "default"
            context: null
            deletePodAfterCompletion: true
            cpu: "1"
            memory: "512M"
            timeoutMs: 300000
            image:
                registry: "docker.io"
                image: "adevinta/zoe-core"
                tag: null
        local:
            enabled: true
storage: null
secrets: null
expressions: {}
clusters:
    default:
        registry: ${SCHEMA_REGISTRY}
        props:
            bootstrap.servers: ${BOOTSTRAP_SERVERS}
            key.deserializer: "org.apache.kafka.common.serialization.StringDeserializer"
            value.deserializer: "io.confluent.kafka.serializers.KafkaAvroDeserializer"
            key.serializer: "org.apache.kafka.common.serialization.StringSerializer"
            value.serializer: "io.confluent.kafka.serializers.KafkaAvroSerializer"
        groups: {}
        topics:
            topic1:
                name: topic1-long-name

default.env

SCHEMA_REGISTRY=http://localhost:8081
BOOTSTRAP_SERVERS=localhost:9092

[wlezzar]

Hi @rsaltrelli : ) . Have you checked out this? https://adevinta.github.io/zoe/configuration/chain/ . You can use the ZOE_CONFIG_OVERRIDE environment variable to override part of zoe's configuration. Does it suit your use case? If not can you explain me in a bit more detail your use case?

[G3zz]

Seconded, I found that the config override merge seemed to remove keys from the objects (i.e., in the example above, if I specified props.bootstrap.servers using ZOE_CONFIG_OVERRIDE the values for key.deserializer, value.deserializer, key.serializer and value.serializer would disappear), meaning I had to re-specify them in the ZOE_CONFIG_OVERRIDE variable.

Geraint

[rsaltrelli]

This seems like it should work for simple uses cases. If what @G3zz says is true and it forces you to redefine that entire section of the YML using env vars then that is a bit inconvenient.

[wlezzar]

Ok I see. Indeed the config override has the disadvantage that it doesn't work well with nested arrays and objects.

I will iterate on this and open a PR.

[wlezzar]

I did a first iteration on this feature and opened a PR here: https://github.com/adevinta/zoe/pull/14

You can find an example in the PR.

I still need to do some testing but you can already provide me with some feedback : ) . Also, is the documentation describing this feature clear enough?

[wlezzar]

@rsaltrelli @G3zz The feature is now available on the latest version v0.26.0