Scan IDs, filenames, saved files, rb numbers, archiving, etc

[Tom-Willemsen]

As a data administrator (?) I would like to figure out how scans output data gets stored in the long term.

bluesky output files contain scientifically-useful data, grouping together a series of related measurements into a coherent "dataset". Therefore they should probably be able to be stored long-term in a similar way to the existing .nxs files.

@GRyall and @ChrisM-S have already both told me they have opinions on this topic.

Acceptance criteria

Documented resolution (and implementation tickets) for the following:

• Figure out what our file naming scheme is for bluesky scans, whether we use GUID or scan_id in various combinations with/without instrument name prepended and so on.
• Figure out where bluesky can write data so that it is treated as long-term scientifically-useful data and gets archived in the right ways for long-term scientifically-useful data.
• Do downstream systems need to associate these files with an RB number? How does that work and what metadata do we need to add to our files to help downstream systems?

Notes

• A bluesky "scan" has no defined relationship to a DAE run.
  • It may contain zero or one or many DAE runs.
• It is reasonable to assume that a scan will sit within a single RB number and a single cycle.
• Bluesky has the concept of a scan_id - a numeric ID. By default this is not persisted anywhere and simply starts from 1 and increments in each new python session, but it is easy to configure the run engine to get the "next scan id" from any source (e.g. a file, database, network service, etc etc). If we want a constantly-incrementing scan id on a per-instrument basis that is easy to do.
• Bluesky also assigns a globally-unique ID to each scan.
  • NSLS2 use GUIDs for all scan data and don't use a scan_id at all, as far as I understand it.

[jackbdoughty]

Reflectometry scientists, at least, would like axis_date_time formatted filenames - therefore not using scan_ids.

They would like them in a cycle-number specific folder (the cycle number is available as a PV from the DAE)