search

awslabs / aws-glue-schema-registry

AWS Glue Schema Registry Client library provides serializers / de-serializers for applications to integrate with AWS Glue Schema Registry Service. The library currently supports Avro, JSON and Protobuf data formats. See https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html to get started.
Apache License 2.0
129 stars 96 forks source link

Support environment variables for Kafka Connect config #46

Open oscarkaypee opened 3 years ago

oscarkaypee commented 3 years ago

Hi all,

We're using the latest Glue Schema Registry source with our dockerized Kafka Connect set up which uses environment variables for it's configuration https://docs.confluent.io/platform/current/installation/docker/config-reference.html#kconnect-long-configuration

This in theory means properties such as value.converter.avroRecordType=GENERIC_RECORD would be represented as CONNECT_VALUE_CONVERTER_AVRORECORDTYPE: GENERIC_RECORD in our environment variables for the container, following the Confluent docs above. However, this runs into: [2021-05-24 12:00:31,125] INFO avroRecordType key is not present in the configs {avrorecordtype=GENERIC_RECORD, schemaautoregistrationenabled=true, region=eu-west-1} (com.amazonaws.services.schemaregistry.common.configs.GlueSchemaRegistryConfiguration)

I've tried:

Which all result in the same lowercase avrorecordtype in the configs e.g. {avrorecordtype=GENERIC_RECORD}

Is it possible to support setting these properties via environment variables? Perhaps by ignoring case in the presence check?

LinyuYao1021 commented 3 years ago

Hi,

Thanks for reaching out. Our configuration parameters are case sensitive. We will look deeper to see if we can support for insensitive ones.

WhiskyTangoFawks commented 3 years ago

I would like to bump this issue. I ran into the same problem- the confluent docker container creates the configs with all lower case, so it's impossible to use this library with them without hacking a workaround.

mohitpali commented 3 years ago

While i am looking into this, would you mind describing your use case ?

WhiskyTangoFawks commented 3 years ago

During startup, the confluent docker containers create all the config files, based on env variables (set in the docker compose). You can add the aws-registry variables to the env variables, and confluents scripts will put them in the config files, but they're all in lower case. I'm doing a spike, and looking at using a containerized kafka connect with aws glue. What I would like to do is be able to configure it through the docker compose env variables, and let the config scripts create the config files without having to either alter the library or script. Having it accept either an all lower case, or be case insensitive, would solve the issue.

feigi commented 2 years ago

Hi, having this issue too. Trying to pass on config params using docker-compose environment variables:

kafka-connect:
    image: confluentinc/cp-kafka-connect:latest
    ...
    environment:
      ...
      # these 2 values currently rely on a custom fix in the aws library due to a case conversion issue
      CONNECT_VALUE_CONVERTER_SCHEMAAUTOREGISTRATIONENABLED: True
      CONNECT_VALUE_CONVERTER_AVRORECORDTYPE: "GENERIC_RECORD"
cfrommert commented 11 months ago

Any updates on this? It would be nice to set the ENV vars for a ECS service.