xpdAcq / rapidz

Fork of Streamz
https://xpdacq.github.io/rapidz
BSD 3-Clause "New" or "Revised" License
5 stars 7 forks source link

add state reset to accumulate #17

Closed CJ-Wright closed 5 years ago

CJ-Wright commented 5 years ago

During the operation of a pipeline it may be necessary to clear the state in an accumulate node or restore the initial state. For instance during a temperature scan we also perform an inner scan of summing images until a threshold is met. This will require us to clear the accumulate state once the threshold is met and the next set of images are acquired. One way to handle this (which is currently the only way to handle this) is to have a second node which resets the state of the accumulate node. This is particularly problematic because it is essentially "spooky action at a distance" (coin credit to Tom). The DAG does not track this at all since it has no formal relationship to the accumulate node. This means that any storage of the minimal DAG for this node (which only includes ancestors) will miss this critical piece of code.

I propose potential solutions to this:

  1. Have a sentinel data type which when received by the accumulate node causes the node to revert to the initial state.
  2. Have a secondary input connection which when emitted into causes the accumulate note to revert to the initial state.

I'm currently leaning towards 2, since I dislike making sentinels if I can avoid them, since the boil down to a special datatype which we need to worry about. Additionally emitting into a secondary stream which causes the reset of the accumulate node doesn't require us to inspect the data so this is automatically parallel safe.

This PR implements 2. This may form a generall approach for these kind of state storing nodes (eg. sliding_window or zip)

CJ-Wright commented 5 years ago

This is failing because pytest and pytest-tornado aren't playing nice together.

codecov-io commented 5 years ago

Codecov Report

Merging #17 into master will not change coverage. The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master      #17   +/-   ##
=======================================
  Coverage   88.84%   88.84%           
=======================================
  Files          16       16           
  Lines        1757     1757           
=======================================
  Hits         1561     1561           
  Misses        196      196

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 3ef03d5...b021d8a. Read the comment docs.