xpdAcq / mission-control

Releases, Installers, Specs, and more!
0 stars 4 forks source link

Run Tomography as Server #159

Closed CJ-Wright closed 5 years ago

CJ-Wright commented 5 years ago

We can run tomographic reconstruction as a RunRouter server. This would solve a rather nasty problem we are facing with the current linking system. Currently the linker will trip over itself, as all the results of the tomo pipeline are nodes named tomo and the input is named qoi. This causes the namespace to overwrite itself. A mutating linker was created to fix this, but it is a rather janky fix and requires a ton of foreknowledge about what kind of data is being dealt with.

A better fix would be to have a RunRouter server. We would then submit any and all analyzed/raw data to the server. The server would then decide if the data meets the criteria for tomo reconstruction. If so it would reconstruct the data and pass it to any interested parties. Else it would do nothing. This makes the system very flexible, as we can decide at runtime if the data is of interest.

There are some technical hurtles for this. The biggest is to figure out what the incoming data will look like, as we need elements from both the raw data and processed data, but maybe we should roll those into one data stream via AlignEventStreams (note that this requires raw data stripping that we don't have currently, but need for other things too). Ideally we would ingest from the two streams at once and run the strip/align parts on the inside of the server to prevent duplication of effort but that may require a zmq ingestor for rapidz similar to the kafka system in place.

CJ-Wright commented 5 years ago

Attn: @davesprouster

CJ-Wright commented 5 years ago

March ish full field