Revisiting JSON handling and workflow interface

[SimonHeybrock]

Previous assumptions

• JSON-Nexus in run-start- message, which gets parsed into a nested dict.
• Insert updates as NXlog, NXevent_data into nested dict.
• Pass parsed JSON to reduction workflow
• Workflow reads, e.g., NXdetector (containing NXevent_data), NXmonitor (containing NXevent_data).

The list item is linked to a parallel development in workflows, now in ESSreduce, where we avoid loading the entire file but just individual groups.

New developments

• JSON-Nexus will not contain everything, since geometry is too large.
• Instead we will get a reference to a Nexus template file.

Question

Should we read NXdetector from Nexus template file, and pass only NXevent_data from beamlime to workflows? That is, further split the loaders in the workflows to load NXevent_data separately?

• Beamlime would become simpler, since it does not have to recursively assemble nested dict. Only make NXevent_data(s) from flatbuffer message(s).
• Workflows would have to read NXdetector without events from file...
  • Implies that workflows needs to be initialized with a filename.
• ... and with every update, read just new events from NXevent_data (which it then can combine with the event-less NXdetector, grouping by pixel if desired).

[SimonHeybrock]

Trying this out on the SANS workflow. From what I can tell so far the proposal makes sense, and even leads to some improvements.

Before:

After:

Notes:

• Some functions and types may need renaming now, I left it for making before/after comparison easier.
• By getting the detector IDs from the RawData[SampleRun], which is now the "empty" detector with no events masks can be loaded and processed without loading potentially huge amounts of events. Even if the events will be loaded anyway this will improve parallelism, and will avoid reprocessing of masks if the event data changes.
• Chunking loading of the event data (using Sciline's map-reduce) with the "after" graph will only load new events. Other detector information as well as masks can be reused. In short, the graph is "wider", which is better.

My conclusion is that we should go ahead as discussed/proposed. Other thoughts @MridulS @YooSunYoung?

[SimonHeybrock]

I made changes in scipp/essreduce#53 and scipp/esssans#153 as discussed. @YooSunYoung we should probably have a chat about changing the workflow interface?