APS-USAXS / bluesky

USAXS Bluesky instrument for APS-U era
0 stars 0 forks source link

refactor data reduction to use APS Data Management #5

Open prjemian opened 7 months ago

prjemian commented 7 months ago

Currently, preliminary data reduction is managed by local Python processes running via cron tasks. These processes could be executed as APS Data Management workflows on demand from bluesky, as a run is operating or afterwards as the team decides.

Integration of data processing with DM has two principle parts:

Due to the differences in the type of data acquired, we'll need separate workflows and data processing for each of these types of measurement:

The code for these steps is available but probably needs adaptation to this new strategy:

Decisions will need to be made regarding where raw and processed data is stored (local or Voyager) and the formats (probably Bluesky databroker and HDF5). Needs consult with teams from USAXS and APS DM. Then implement the decisions in bluesky controls.

prjemian commented 7 months ago

Existing python code needs update to python 3