search

bluesky / bluesky-queueserver

Server for queueing plans
https://blueskyproject.io/bluesky-queueserver/
BSD 3-Clause "New" or "Revised" License
11 stars 20 forks source link

Provide authorisation/security information to kafka producer. #256

Open stmudie opened 1 year ago

stmudie commented 1 year ago

Expected Behavior

Our Kafka servers are secured using SSL and SASL. We wish to supplying the required parameters to the Queueserver's in built Bluesky-Kafka Publisher instance (via the producer_config) for connecting to our servers.

Current Behavior

Currently there is no way of passing this information through to the Bluesky Kafka Producer instance provided by default within the Queueserver.

Possible Solution

Add a command line argument to start-re-manager for passing a path to a Kafka configuration file then reading said config file and passing through to the Kafka-Bluesky Publisher constructor. For example: https://github.com/stmudie/bluesky-queueserver/compare/783b950..895d150 I'm happy for this code to form the basis of a PR (after updating to bring in latest commits from upstream).

Context

Without this configuration option (or an alternative method for configuring the subscribed Kafka publisher/producer) we can't use the Queueserver with our standard Kafka deployments.

dmgav commented 1 year ago

While it is possible to extend the set of command line arguments, I don't think it is the best way to go, since it is unlikely we could cover every possible scenario. It is likely that built-in options to create Run Engine and subscribe it to Data Broker or Kafka Publisher are going to be used only for demos and tutorials. In production deployments we create an instance of Run Engine (it must be a variable named RE) in the startup script and add all needed subscriptions. If start-re-manager is passed the parameter --keep-re then no additional instance of RE Engine is created and Run Engine defined in the startup code is used by the Queue Server to run plans.

tacaswell commented 1 year ago

I'm also inclined to think that --keep-re is the right way to go short term. I am coming to think my initial push to have QS create its own RE was a mistake.

Medium term, it might also be worth investigating something like what tiled does with handlers to have a config file that specifies what callbacks to attach the RE that it creates rather than passing this configuration through the command line.

dmgav commented 1 year ago

In production we set all callbacks in the startup code. I think that may be the default mode should be the one that uses Run Engine created in the startup scripts and the other mode should be explicitly activated. Also we don't want to couple Queue Server deployment and configuration of other services not used by the Queue Server directly, such as Kafka.