xpdAcq / mission-control

Releases, Installers, Specs, and more!
0 stars 4 forks source link

Pipeline dispatch #131

Open CJ-Wright opened 5 years ago

CJ-Wright commented 5 years ago

It would be great if:

  1. We had multiple pipelines which we could choose from (eg a ctPDF, standard pdf/xrd, auto acq, autonomous motor, etc.)
  2. We could inspect the start document and decide which pipeline to use (based off some metadata we insert into the start document)
  3. A dispatcher (on the analysis side) decides which pipeline to use and either uses an existing instantiated pipeline or instantiates a new pipeline to handle the data.
  4. Data is then pushed into the pipeline returned by the dispatcher.

Expected Behavior

The analysis pipeline knows what kind of analysis we want based off of the start document and runs it.

Current Behavior

There is only one pipeline.

Possible Solution

  1. Have pipelines created by pipeline factory functions
  2. Write a simple dispatcher which inspects the 'analysis_type' key in the start document and returns an instance of the associated pipeline (store pipeline factories in a dict)
CJ-Wright commented 5 years ago

A potential alternative to this would be to have a "smart" linking system that would link/delink pipelines as needed. For example we are running a standard pipeline, and then switch to auto exposure. The linker would need to:

  1. delink the raw source from the input data
  2. insert an accumulate node between the raw source and the raw light data
  3. insert a series of nodes to decide if the exposure is done

A similar process would be needed to go back to the standard pipeline.

This could be a bit tricky but possible. The main benefit is that anything cached in the pipeline is still cached in the pipeline for other use cases.

CJ-Wright commented 5 years ago

See: https://github.com/NSLS-II/bluesky/pull/1076

CJ-Wright commented 5 years ago

The rewrite associated with this change (making pipelines build on the fly) will have such broad consequences that I'm turning this into an epic so we can track all the things it touches.