NSLS-II / nslsii

NSLS-II related devices
BSD 3-Clause "New" or "Revised" License
10 stars 21 forks source link

Support new sections in Kafka config file #160

Closed jklynch closed 1 year ago

jklynch commented 1 year ago

This PR proposes support for handling new sections in the Kafka config file (/etc/kafka.yml by default). The new sections are currently optional but are expected to become required in the future. Two of the new sections are actually ignored, but have some test coverage.

The new sections specify shared security configuration for producers and consumers, limited consumer-specific configuration, and a list of topic templates for RunEngine producers.

This is an example of a configuration file with all optional sections:

---
  abort_run_on_kafka_exception: false
  bootstrap_servers:
    - localhost:9092
    - kafka2:9092
    - kafka3:9092
  runengine_producer_config:
    compression.codec: snappy
    security.protocol: SASL_SSL
    sasl.mechanisms: PLAIN
    ssl.ca.location: /etc/ssl/certs/ca-bundle.crt
  producer_consumer_security_config:
    security.protocol: SASL_SSL
    sasl.mechanisms: PLAIN
    ssl.ca.location: /etc/ssl/certs/ca-bundle.crt
  consumer_config:
    auto.offset.reset: latest
  runengine_topics:
    - "{endstation}.bluesky.runengine.documents"
    - "{endstation}.bluesky.runengine.{document_name}.documents"

The last three sections, producer_consumer_security_config, consumer_config, and runengine_topics are new and optional. The last two are currently ignored by nslsii.configure_base.

In addition, this PR simplifies the docker-compose script for running Kafka in a container.