Connect to Kafka with sasl jaas config

[silarr]

Currently there is a way to pass producer and consumer config to CDC via environment parameters like: EVENTUATE_LOCAL_KAFKA_CONSUMER_PROPERTIES_SASL_JAAS_CONFIG and EVENTUATE_LOCAL_KAFKA_PRODUCER_PROPERTIES_SASL_JAAS_CONFIG But it is not enough to make CDC server working. One of the reason is, probably, KafkaHealthCheck which only accepts Kafka servers parameter to build it's Kafka consumer. This causes an exception java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access.

Another way of passing parameters to configure consumers would be via something similar to this JAVA_OPTS: "-Djava.security.auth.login.config=/opt/cdc/kafka/conf/kafka_jaas.conf". This would make an exception with Zookeeper: SASL configuration failed: javax.security.auth.login.LoginException: Zookeeper client cannot authenticate using the 'Client' section of the supplied JAAS configuration: '/opt/cdc/kafka/conf/kafka_jaas.conf' because of a RuntimeException: java.lang.SecurityException: java.io.IOException: Configuration Error: Line 1: expected [{], found [request.timeout.ms] Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.

This may not be an issue by itself but could be an indication of CDC's Zookeeper coupling with Kafka's Zookeeper. Since the new version of Kafka removes Zookeeper - it would be great to be able to configure CDC to use Zookeeper or Redis, etc for leadership election without the knowledge if CDC uses Kafka or Redis, etc at all.

[silarr]

The easiest way to test that CDC works with secure Kafka with hidden (or separated/removed) Zookeeper would probable be to use Confluent Kafka in their Cloud (https://confluent.cloud/). It would be about free to use for testing of this use case.

[cer]

https://eventuate-users.slack.com/archives/C3L9MNMQC/p1591350207110100

[g-portugues]

In the environment variables of the cdc-service container:

    environment:
      JAVA_OPTS: |
        -Deventuate.local.kafka.producer.properties.security.protocol=SASL_SSL
        -Deventuate.local.kafka.producer.properties.sasl.mechanism=PLAIN
        -Deventuate.local.kafka.producer.properties.client.dns.lookup=use_all_dns_ips
        -Deventuate.local.kafka.producer.properties.session.timeout.ms=45000
        -Deventuate.local.kafka.producer.properties.acks=all
        -Deventuate.local.kafka.consumer.properties.security.protocol=SASL_SSL
        -Deventuate.local.kafka.consumer.properties.sasl.mechanism=PLAIN
        -Deventuate.local.kafka.consumer.properties.client.dns.lookup=use_all_dns_ips
        -Deventuate.local.kafka.consumer.properties.session.timeout.ms=45000
        -Deventuate.local.kafka.consumer.properties.acks=all
        -Djava.security.auth.login.config=/tmp/kafka/jaas.conf
...