Better dark handling

[CJ-Wright]

Taking dark images (darks) is an important part of the data acquisition, especially PDF data where the signal is weak.

Current implementation: Currently we take a dedicated dark scan at the beginning of the scan if the time between the previous dark and now is greater than the dk_window. That scan is then linked to subsequent scans via a foreign key which the data analysis tools know how to access and look up.

Problems with the current implementation:

1. A foreign key might be a bit too weak. It might be stronger to directly associate the data.
2. This system doesn't take darks in the middle of long scans. For a 12 hour scan you might get a dark at the start but at no other time. This could cause problems as the dark drifts.

Solution: Always have darks on hand in a separate stream. This will remove the foreign key issue and make all the scans uniform.

Proposed implementation:

1. Change the detector so it has a cache ({(acq_time, det_params): datum}) for datums containing darks.
2. When we want a dark, provide the detector with the current dark timeout. If there is a dark in the cache which is inside the dark timeout window and has the correct parameters, return something which says that the dark is available. Otherwise say that we need to take a dark. An implementation of this would be a helper method to inspect the dark cache def check_dark_cache(self, timeout).
3. If a dark is not available close the shutter
4. Trigger and read the detector specifying that we are taking a dark. If a dark was available it will return a reference to the prior dark (Note that in the event that multiple darks are viable (eg we loosened the timeout) choose the latest one), otherwise it will take a dark and cache it.
5. Always trigger a dark at the beginning of every scan.
6. Optionally trigger a dark every time the dark window is out of date. This could take the form of a plan mutator which changes trigger and read to check the dark then do a dark trigger read.

The pipeline will know how to handle the dark images by looking at the dark stream.

This also nicely supports dynamic dark acquisition, as the pipeline could inspect the images and decide if a new dark is needed and forcing a dark image.

attn @DanOlds @tacaswell

[DrMorpheus]

There are two points I'd throw into this mix.

1) There needs to be a delay time to account for the time it takes for the shutter to actually close. In talking with @DanOlds yesterday, the time it takes for the shutter to open and close can vary (it shouldn't, but does), with the slower end of the range taking up to 5 seconds.

2) There also needs to be an additional delay once the shutter is closed to allow the detector to clear any residual signal before the dark collection begins. We've found this to be quite critical, as oftentimes in our data we see that the dark patterns will still have diffraction features present if the shutter is not closed for a sufficient amount of time.

My two cents have been added. I hope this helps the implementation discussion.

M

[CJ-Wright]

@DrMorpheus Thank you very much for the input!

1) This really needs to be handled on the shutter hardware/epics side, as discussed in https://github.com/xpdAcq/mission-control/issues/154 Currently when we wait for the shutter to finish its motion it tells us that it is done without actually having finished the motion. While we have implemented a temporary fix in the most recent release of xpdAcq this fix is rather detrimental long term and consists of a level of magic that will eventually cause problems.

2) I think the dynamic dark acquisition, which could amount to taking darks until the detector is "clean", that this approach would enable could help solve that issue.

[DanOlds]

Something new we noticed today - the dark-logic does not check if the same detector is being used to measure the light that is being used for a cached dark. This is to say, if the dark time is large, and a user switches from taking images on pe2 to pe1, the incorrect dark is subtracted from the image.

Ideally, the dark-logic would verify that the associated detector is correct at the time of checking if a new dark is needed, and trigger a new dark if this has changed.

This happened last night on XPD, and caused some data loss.