Enhancement proposal: allow configuration of sasl.username and sasl.password from environment variables.

[odinnou]

Hello, first of all, great lib, it works like a charm!

But, I'm facing an issue for deployment on Kubernetes, I need to configure the username and password to access to the Kafka cluster and I can't rely on volume to do that. Indeed all the configuration must be transmit through environment variables specified on the deployment file.

It could be helpfull, if you give the option to override all the consumer/producer configuration keys from environment variables (and not only the bootstrap servers and consumer group).

Thanks in advance, kind regards Rémi.

[fkarakas]

Hello @odinnou, Thank you very much for your nice comment, and it is nice to see that the kafka message scheduler is useful for someone :) Regarding the configuration of the kafka consumer and producer in the scheduler, I think it is better to use the "new" yaml configuration file (thanks to @james-johnston-thumbtack), there is plenty of configuration available in the confluent library and we cannot create an environment variable for each of them (see https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)

This is an example of kubernetes manifests and how you can create a file in your pod from a config map, it is very straightforward.

apiVersion: v1
kind: ConfigMap
metadata:
  name: env-config
  namespace: default
data:
  BOOTSTRAP_SERVERS: localhost:9092
  CONFIGURATION_FILE: /app/config.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: yaml-config
  namespace: default
data:
  config.yaml: |
    kafka.common.configuration:
        sasl.mechanisms: PLAIN
        sasl.username: username
        sasl.password: password
        security.protocol: SASL_SSL
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-message-scheduler
  namespace: default
  labels:
    app: kafka-message-scheduler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-message-scheduler
  template:
    metadata:
      labels:
        app: kafka-message-scheduler
    spec:
      containers:
      - name: kafka-message-scheduler
        image: etf1/kafka-message-scheduler:v0.0.6
        envFrom:
        - configMapRef:
            name: env-config
        volumeMounts:
        - name: config-volume
          mountPath: /app/config.yaml
          subPath: config.yaml        
        ports:
        - containerPort: 8000
      volumes:
      - name: config-volume
        configMap:
          name: yaml-config

please let me know if you have any issues or questions.

Regards

[odinnou]

Hello, thank you for this configuration file, it helps me a lot!