search

NSLS-II / lsdc

LSDC - Life Sciences Data Collection
https://nsls-ii.github.io/lsdc
BSD 3-Clause "New" or "Revised" License
3 stars 14 forks source link

Provide Kafka hooks for processing and other systems #64

Open JunAishima opened 3 years ago

JunAishima commented 3 years ago

This will allow better abstraction of processing from LSDC itself.

Requirements:

  1. Kafka cluster must be up and accessible from AMX/FMX - needs investigation
  2. Decisions need to be made on channel names and content that are consistent with DAMA standards
  3. Decisions need to be made on what content is required for processing and whatever else people want to do with these hooks
  4. Development of basic client code that will allow staff to develop the functionality they need.
  5. Development of conda environments for server that include the confluent-kafka conda
    • May require methods to monitor who is using the channels
jklynch commented 3 years ago

@JunAishima there is a kafka cluster running on kafka1.nsls2.bnl.gov, ..., kafka7.nsls2.bnl.gov. It is visible to all beamlines. You will need some new topics for the MX beamlines. Garrett and I know how to add them and we can show you how.

JunAishima commented 3 years ago

thanks, @jklynch - this work is being delayed for higher priority work. I will talk with you or Garrett to register a topic, I wasn't sure whether I needed to or not!