Add more explanation of circular buffer

[ianhi]

I just saw @marktsuchida's comment in #168 and am extremely curious.

Also, before anybody gets confused, the current "circular buffer" is not a circular buffer or ring buffer. It is actually a bounded queue. Except in Live mode (where overflow is allowed), where it is completely weird.

From https://github.com/micro-manager/mmCoreAndDevices/blob/4edb2c970793fffcbb3c9fa906b710d8feb694d0/MMCore/MMCore.cpp#L2972-L2976

I was treating it as such. I'm particular curious about this for any implications it may have for a good live mode. e.g. we used to trying to display everything in the buffer until we figured out to just getLastImage: https://github.com/tlambert03/napari-micromanager/pull/40

[marktsuchida]

The only real reference is what MMStudio does, but here is an overview for a single camera. It is also good to observe the behavior in Live vs MDA using the Sequence Buffer Monitor plugin of MMStudio.

• Sequence acquisition (or anything intended as scientific data) should be started with the stopOnOverflow parameter set to true. This means that no frames are discarded (modulo bugs in the device adapter or driver). The sequence buffer works like a bounded queue, bounded by the buffer size. If the buffer fills up, the acquisition fails and stops. The client should retrieve images using one of the popNext*() functions, which always retrieve the oldest frame in the buffer.

  • One could argue that this is a "circular buffer", because the memory is pre-allocated and recycled, in order, in the current implementation. But it is not a contiguous buffer (as the term would normally imply; in the early days of MM it was contiguous), and the order of recycling is an implementation detail -- it would in fact be better for memory locality if we reused the most recently popped buffer (for frame sizes small enough to fit in the CPU cache).

• Live mode should be started with stopOnOverflow set to false and the client should retrieve images using one of the getLast*() functions, which always retrieve the newest frame in the buffer. Those functions do not remove the image from the sequence buffer, which therefore fills up over time. When the sequence buffer is full, the device adapter (not the Core!) clears the entire sequence buffer, and starts filling it again. See, e.g., https://github.com/micro-manager/mmCoreAndDevices/blob/4edb2c970793fffcbb3c9fa906b710d8feb694d0/DeviceAdapters/DemoCamera/DemoCamera.cpp#L1118-L1125 (This code runs on a thread created by the device adapter or driver, depending on the camera.)

You might think that, in Live mode, it would be simpler if the Core just kept the latest image, never allowing the buffer to fill up. You'd be right in the single camera case. But we can't make that isolated change due to how Multi Camera works.

In addition to Multi Camera, we are also constrained by the desire to keep the Live Replay plugin, which is supposed to allow the user to view what just happened. It simply pops all the images left in the sequence buffer after an immediately preceding Live.

• One problem that should be obvious from the above description of sequence buffer behavior is that the number of images that Live Replay retrieves is up to your luck -- you get very few if the sequence buffer was cleared just before Live was stopped.
• Another, more subtle problem, is that some cameras change their behavior according to the stopOnOverflow flag, such that they may internally drop frames (i.e., switch to a best-effort mode) or acquire at a non-uniform frame rate (via software triggering). This could be bad for any scientists who expect to get real data out of Live Replay.

We could probably move the clear-on-overflow behavior into the Core and change it so that it only removes the oldest frame to make space. This should be compatible with existing camera adapters (which will just never see an overflow), with Multi Camera, and with Live Replay (and partially improve Live Replay behavior).

So what about multiple cameras? The Multi Camera device adapter simply forwards StartSequenceAcquisition() calls to each of the physical cameras. The physical cameras send their images via the normal mechanism, shown above for DemoCamera. So images from different cameras end up in the same sequence buffer, with no guarantee of ordering (in fact, there is no requirement that the two cameras be acquiring at exactly the same frame rate, at least in the case of Live mode).

• In MDA, MMStudio (via acqEngine) pops every image that gets placed in the sequence buffer. Since the images are tagged with the camera number or name, they get placed into different channels in the downstream datastore and viewer (possibly after combining (Cartesian product) with the channels of the MDA settings). This all works because the sequence acquisition is finite; it is guaranteed that each physical camera will generate the same number of images.

  • But it could induce various bugs if people write code that expects images to arrive in some particular channel order.

• In Live mode, where we cannot retrieve every image from MMCore (for performance reasons), it gets nasty. The SnapLiveManager implementation looks at up to 6 times the number of physical cameras of the latest images in the sequence buffer by calling getNBeforeLastTaggedImage() (which is MMCoreJ's wrapper of getNBeforeLastImageMD()). It then hopes that the latest frame from each physical camera is present in that range.

  • For high frame rates, the window of the last N images will change even while this is happening, meaning that each iteration of getNBeforeLastTaggedImage() is not guaranteed to retrieve distinct frames, which is why the factor of 6 is used to increase the chance of encountering frames from every camera (otherwise you would think that a factor of 2 or so would be enough). Still, the behavior is unreliable, as you can tell by imaging the worst case scenario where the loop is running very close to the rate of images arriving and therefore keeps retrieving the same-ish frame.
  • Even for low frame rates, nothing guarantees that corresponding frames are retrieved from the different cameras, even if the cameras are operating in lockstep (possibly via TTL synchronization). This matters less to the human eye for high frame rates (provided that the display is operating fast enough), but could cause issues for specimens that move or change fast compared to a low frame rate (this includes moving the XY stage or focusing).

Hopefully this illustrates the sorts of issues we face if we try to make backward-compatible improvements to the API. This is why in #168 I proposed that we should add a completely new interface and provide backward compatibility (if at all) through emulation (many years ago, I even started writing some code in that direction, although my approach at the time was probably too ambitious, especially with pre-modern C++). In addition to everything above, we are also constrained by the MMDevice interface (where backward compatibility is even more crucial), but I think that interface has fewer problems in this particular area.

[ianhi]

Thanks mark! This is wonderfully detailed and very helpful. One follow up question. I noticed the MM livemanager that the seqeuenceacquisition timing is set to 0:

https://github.com/micro-manager/micro-manager/blob/4a5d51ea76f89eaa6e74d0d772822701661aa76a/mmstudio/src/main/java/org/micromanager/internal/SnapLiveManager.java#L256

Any insight and why it's that way rather than the exposure value?

We've been setting it to be the current value of the exposure:

https://github.com/tlambert03/napari-micromanager/blob/157ec63eebb3d370d89f3fecef9025cd33e6333d/micromanager_gui/main_window.py#L222

and then stopping and restaritng the acquistion if the exposure changes: https://github.com/tlambert03/napari-micromanager/blob/157ec63eebb3d370d89f3fecef9025cd33e6333d/micromanager_gui/main_window.py#L318-L321

is that approach worse somehow? Or is perhaps precluded by the multi-camera support in micromanager?

[marktsuchida]

The startContinuousSequenceAcquisition() (which is what Live actually uses, despite what I wrote above) is like startSequenceAcauisition() except that numImages is set to infinity and stopOnOverflow is forced to false.

(Unfortunately this is left to individual device adapters, but the correct thing for them to do is what DemoCamera does: https://github.com/micro-manager/mmCoreAndDevices/blob/4edb2c970793fffcbb3c9fa906b710d8feb694d0/DeviceAdapters/DemoCamera/DemoCamera.cpp#L1048-L1051 (whether LONG_MAX is correct here depends on the camera).)

With all variants of start*SequenceAcquisition(), the intervalMs parameter is ignored (please don't shoot the messenger :). To be more precise, it is ignored by all (I hope, for consistency's sake) camera device adapters. I don't know the early history of this, but I suspect nobody bothered to actually implement the behavior in early camera adapters, and therefore MMStudio never bothered to pass anything other than 0, and therefore nobody has bothered to implement it in camera adapters. (Some cameras that support setting a frame rate independent of exposure do so through a custom property. This makes more sense anyway, other than the lack of standardization.) At this point, we should probably rename it unused, ignore what the app passes, and always pass 0 to device adapters.

As for changing the exposure during a live acquisition, yes, it is correct to restart the acquisition. You have to stop the acquisition, set the exposure, then start a new acquisition, because many cameras are incapable of changing the exposure while an acquisition is running. There has been talk about allowing the change to be applied without restarting for cameras that opt in.