Open ianhi opened 2 years ago
The only real reference is what MMStudio does, but here is an overview for a single camera. It is also good to observe the behavior in Live vs MDA using the Sequence Buffer Monitor plugin of MMStudio.
Sequence acquisition (or anything intended as scientific data) should be started with the stopOnOverflow
parameter set to true
. This means that no frames are discarded (modulo bugs in the device adapter or driver). The sequence buffer works like a bounded queue, bounded by the buffer size. If the buffer fills up, the acquisition fails and stops. The client should retrieve images using one of the popNext*()
functions, which always retrieve the oldest frame in the buffer.
Live mode should be started with stopOnOverflow
set to false
and the client should retrieve images using one of the getLast*()
functions, which always retrieve the newest frame in the buffer. Those functions do not remove the image from the sequence buffer, which therefore fills up over time. When the sequence buffer is full, the device adapter (not the Core!) clears the entire sequence buffer, and starts filling it again.
See, e.g.,
https://github.com/micro-manager/mmCoreAndDevices/blob/4edb2c970793fffcbb3c9fa906b710d8feb694d0/DeviceAdapters/DemoCamera/DemoCamera.cpp#L1118-L1125
(This code runs on a thread created by the device adapter or driver, depending on the camera.)
You might think that, in Live mode, it would be simpler if the Core just kept the latest image, never allowing the buffer to fill up. You'd be right in the single camera case. But we can't make that isolated change due to how Multi Camera
works.
In addition to Multi Camera
, we are also constrained by the desire to keep the Live Replay plugin, which is supposed to allow the user to view what just happened. It simply pops all the images left in the sequence buffer after an immediately preceding Live.
stopOnOverflow
flag, such that they may internally drop frames (i.e., switch to a best-effort mode) or acquire at a non-uniform frame rate (via software triggering). This could be bad for any scientists who expect to get real data out of Live Replay.We could probably move the clear-on-overflow behavior into the Core and change it so that it only removes the oldest frame to make space. This should be compatible with existing camera adapters (which will just never see an overflow), with Multi Camera
, and with Live Replay (and partially improve Live Replay behavior).
So what about multiple cameras? The Multi Camera
device adapter simply forwards StartSequenceAcquisition()
calls to each of the physical cameras. The physical cameras send their images via the normal mechanism, shown above for DemoCamera
. So images from different cameras end up in the same sequence buffer, with no guarantee of ordering (in fact, there is no requirement that the two cameras be acquiring at exactly the same frame rate, at least in the case of Live mode).
In MDA, MMStudio (via acqEngine) pops every image that gets placed in the sequence buffer. Since the images are tagged with the camera number or name, they get placed into different channels in the downstream datastore and viewer (possibly after combining (Cartesian product) with the channels of the MDA settings). This all works because the sequence acquisition is finite; it is guaranteed that each physical camera will generate the same number of images.
In Live mode, where we cannot retrieve every image from MMCore (for performance reasons), it gets nasty. The SnapLiveManager
implementation looks at up to 6 times the number of physical cameras of the latest images in the sequence buffer by calling getNBeforeLastTaggedImage()
(which is MMCoreJ's wrapper of getNBeforeLastImageMD()
). It then hopes that the latest frame from each physical camera is present in that range.
getNBeforeLastTaggedImage()
is not guaranteed to retrieve distinct frames, which is why the factor of 6 is used to increase the chance of encountering frames from every camera (otherwise you would think that a factor of 2 or so would be enough). Still, the behavior is unreliable, as you can tell by imaging the worst case scenario where the loop is running very close to the rate of images arriving and therefore keeps retrieving the same-ish frame.Hopefully this illustrates the sorts of issues we face if we try to make backward-compatible improvements to the API. This is why in #168 I proposed that we should add a completely new interface and provide backward compatibility (if at all) through emulation (many years ago, I even started writing some code in that direction, although my approach at the time was probably too ambitious, especially with pre-modern C++). In addition to everything above, we are also constrained by the MMDevice interface (where backward compatibility is even more crucial), but I think that interface has fewer problems in this particular area.
Thanks mark! This is wonderfully detailed and very helpful. One follow up question. I noticed the MM livemanager that the seqeuenceacquisition timing is set to 0:
Any insight and why it's that way rather than the exposure value?
We've been setting it to be the current value of the exposure:
and then stopping and restaritng the acquistion if the exposure changes: https://github.com/tlambert03/napari-micromanager/blob/157ec63eebb3d370d89f3fecef9025cd33e6333d/micromanager_gui/main_window.py#L318-L321
is that approach worse somehow? Or is perhaps precluded by the multi-camera support in micromanager?
The startContinuousSequenceAcquisition()
(which is what Live actually uses, despite what I wrote above) is like startSequenceAcauisition()
except that numImages
is set to infinity and stopOnOverflow
is forced to false
.
(Unfortunately this is left to individual device adapters, but the correct thing for them to do is what DemoCamera does: https://github.com/micro-manager/mmCoreAndDevices/blob/4edb2c970793fffcbb3c9fa906b710d8feb694d0/DeviceAdapters/DemoCamera/DemoCamera.cpp#L1048-L1051 (whether LONG_MAX
is correct here depends on the camera).)
With all variants of start*SequenceAcquisition()
, the intervalMs
parameter is ignored (please don't shoot the messenger :). To be more precise, it is ignored by all (I hope, for consistency's sake) camera device adapters. I don't know the early history of this, but I suspect nobody bothered to actually implement the behavior in early camera adapters, and therefore MMStudio never bothered to pass anything other than 0
, and therefore nobody has bothered to implement it in camera adapters. (Some cameras that support setting a frame rate independent of exposure do so through a custom property. This makes more sense anyway, other than the lack of standardization.) At this point, we should probably rename it unused
, ignore what the app passes, and always pass 0
to device adapters.
As for changing the exposure during a live acquisition, yes, it is correct to restart the acquisition. You have to stop the acquisition, set the exposure, then start a new acquisition, because many cameras are incapable of changing the exposure while an acquisition is running. There has been talk about allowing the change to be applied without restarting for cameras that opt in.
I just saw @marktsuchida's comment in #168 and am extremely curious.
From https://github.com/micro-manager/mmCoreAndDevices/blob/4edb2c970793fffcbb3c9fa906b710d8feb694d0/MMCore/MMCore.cpp#L2972-L2976
I was treating it as such. I'm particular curious about this for any implications it may have for a good live mode. e.g. we used to trying to display everything in the buffer until we figured out to just
getLastImage
: https://github.com/tlambert03/napari-micromanager/pull/40