characterize latencies in workflow 3 demonstrated at M4

[prjemian]

At the M4 demonstration of workflow 3 (fully autonomous peak centering), the demo was slow. Describe the sources of latency and what can be done to reduce each. Demonstrate if possible or practical.

[prjemian]

Coarse discussion concluded that latencies might be reduced by factors of 2-5 but BDP is interested in workflows that execute on time scales faster (than workflow3) by orders of magnitude. This means that a workflow based on one acquisition yielding one image file and one analysis will not scale to much faster. We need to switch to a streaming workflow model that avoids file-writing bottlenecks (one of the biggest latencies).

[prjemian]

@sveseli Can you identify the delays added in the DM workflow for the M4 demonstration? Can we control (reduce) any of them?

[sveseli]

DM Workflow

All of those items below are related to the file-based processing:

1. Workflow start. DM DAQ service has a configuration parameter that determines how long to wait to process the file (start the workflow) after it was last written into. This parameter typically differs from beamline to beamline. For BDP, at the moment this is set to 5 seconds and could be reduced (probably to as low as 1 second).
2. Delays in workflow tasks. Although DM workflow engine runs tasks as soon as possible, there may be artificial delays introduced in task scripts and workflow steps. For example-03 workflow, step 03-MONITOR-JOB, we poll SGE every 10 seconds. This could be reduced to something like 1 second. Obviously, in a production environment with hundreds or thousands of files processed simultaneously one must balance speed of workflow execution vs overloading the batch system needlessly.
3. Batch system delays. How busy the system is, are there sufficient resources to run immediately, how long does it takes to schedule and run a job, etc. For BDP, this is not significant. For production environment, it can be a problem.
4. Analysis scripts themselves.

[prjemian]

Bluesky data acquisition

Factors related to 03-workflow (workflow was presented at M4 meeting)​

• Bluesky-queueserver process monitor​

  • Internal poll loop interval set to 1 s.
  • Balances prompt response and server load.
  • Could be faster (0.1 s) at compromise of queueserver resources.​

• Data acquisition algorithm​

  • Many steps for queueserver to process. Individually.
  • Minimum time for each set by queueserver poll loop interval.​
  • Steps could be combined to reduce latency in the queueserver polling loop with loss to flexibility in acquisition sequence.

• Each acquisition step waits for workflow to process and deliver analysis.

  • This is a consequence of file-based workflow designed with move and image acquisition.​

• Summary:

  • At most, 1 order of magnitude improvement possible.
  • Further improvements must come with redesign to streaming workflow with fly-scanning.​