Add Kafka support

[jklynch]

This PR proposes further Kafka support for production and test environments:

• nslsii.configure_base(..., publish_documents_with_kafka=False)
• Kafka configuration is handled with a file
• two options for publishing RunEngine documents with Kafka

By default nslsii.configure_base() will not enable Kafka publishing. In the long term this may change. For now beamlines that want to publish bluesky documents with Kafka will have to set this to True. This gives beamlines a quick and easy way to turn off Kafka publishing if it causes problems.

A configuration file located at /etc/bluesky/kafka.yml is intended to configure Kafka publishing for production. In test environments this path can be overridden using environment variable BLUESKY_KAFKA_CONFIG_PATH.

The configuration file must contain three top-level elements. Here is an example:

---
  abort_run_on_kafka_exception: true
  bootstrap_servers:
    - kafka1:9092
    - kafka2:9092
    - kafka3:9092
  runengine_producer_config:
    acks: 0
    message.timeout.ms: 3000
    compression.codec: snappy

abort_run_on_kafka_exception=true selects the "direct" document publishing scheme, which will abort a run if a Kafka exception occurs. It is anticipated that this will be the production setting in the long term. abort_run_on_kafka_exception=false selects the "queue-thread" document publishing scheme, which will not abort a run if a Kafka exception occurs.

[danielballan]

This hanging test worries me a little. Is it very repeatable that it breaks on 3.7 but not on 3.8? If the tests pass very repeatably on 3.8, then I agree we can let go of 3.7.

[jklynch]

I'm not happy about the hanging test in 3.7 either. This test passes reliably in 3.8 and 3.9. I am looking in to it some more.

[mrakitin]

I would not rush to exclude Python 3.7 from our supported versions, as some private packages are only built to work with Python 3.7 (read getpdfx and friends).

[danielballan]

I took a quick tour but not did perform a line-by-line review. All looks reasonable. Should move ahead.