Acquisition stops temporarily while it catches up with frames

[simonecoppola]

Related to this discussion: https://imagesc.zulipchat.com/#narrow/stream/442785-pymmcore.5B-plus.5D/topic/questions and to the micromanager-gui project as well.

Specifically: "Is it possible to run an acquisition in micromanager-gui without displaying the images? We've noticed that when performing fast acquisitions, often the acquisitions stop at the right time (say, a "fast" 20 or 50ms exposure acquisition for 5s - we see the led come on and off for those 5s) but then it waits until all the images have been displayed before starting on the second channel. We're not too interested in seeing all the images live (maybe the first for each channel would be good for sanity checking the focus is holding, but not too important) so if live display could be disabled, that would be great for us!"

The issue presents itself even if the acquisition is started as a script. I'm attaching here the full console log from the script run! console_log.txt

[tlambert03]

thanks @simonecoppola, pasting the key part of the traceback here:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\..\.virtualenv\Lib\site-packages\psygnal\_signal.py", line 1196, in _run_emit_loop
    self._run_emit_loop_inner()
  File "C:\..\.virtualenv\Lib\site-packages\psygnal\_signal.py", line 1225, in _run_emit_loop_immediate
    caller.cb(args)
  File "C:\..\.virtualenv\Lib\site-packages\psygnal\_weak_callback.py", line 453, in cb
    func(obj, *self._args, *args, **self._kwargs)
  File "C:\..\.virtualenv\Lib\site-packages\pymmcore_plus\mda\handlers\_tensorstore_handler.py", line 204, in frameReady
    self._store = self.new_store(frame, event.sequence, meta).result()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: ALREADY_EXISTS: Error opening "zarr" driver: Error writing local file "Z:\\Shared363\\data\\Coppola-Simone\\0_DATA\\openframe\\240926 PC19 BADA TADA\\test_speed\\console/.zarray" [source locations='tensorstore/internal/cache/kvs_backed_cache.h:208\ntensorstore/driver/driver.cc:112'] [tensorstore_spec='{\"context\":{\"cache_pool\":{},\"data_copy_concurrency\":{},\"file_io_concurrency\":{},\"file_io_sync\":true},\"create\":true,\"driver\":\"zarr\",\"dtype\":\"uint16\",\"kvstore\":{\"driver\":\"file\",\"path\":\"Z:\\\\Shared363\\\\data\\\\Coppola-Simone\\\\0_DATA\\\\openframe\\\\240926 PC19 BADA TADA\\\\test_speed\\\\console/\"},\"metadata\":{\"dimension_separator\":\"/\"},\"schema\":{\"chunk_layout\":{\"read_chunk\":{\"shape\":[1,1,2048,2048]},\"write_chunk\":{\"shape\":[1,1,2048,2048]}},\"domain\":{\"exclusive_max\":[2,100,2048,2048],\"inclusive_min\":[0,0,0,0],\"labels\":[\"c\",\"t\",\"y\",\"x\"]}},\"transform\":{\"input_exclusive_max\":[[2],[100],[2048],[2048]],\"input_inclusive_min\":[0,0,0,0],\"input_labels\":[\"c\",\"t\",\"y\",\"x\"]}}']

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\..\AppData\Local\Programs\Python\Python312\Lib\threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "C:\..\.virtualenv\Lib\site-packages\pymmcore_plus\mda\_thread_relay.py", line 144, in run
    emitter.emit(*args)
  File "C:\..\.virtualenv\Lib\site-packages\psygnal\_signal.py", line 1176, in emit
    self._run_emit_loop(args)
  File "C:\..\.virtualenv\Lib\site-packages\psygnal\_signal.py", line 1213, in _run_emit_loop
    raise loop_err from cb_err  # emit() call ABOVE || callback error BELOW
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\..\.virtualenv\Lib\site-packages\psygnal\_signal.py", line 1196, in _run_emit_loop
    self._run_emit_loop_inner()
  File "C:\..\.virtualenv\Lib\site-packages\psygnal\_signal.py", line 1225, in _run_emit_loop_immediate
    caller.cb(args)
  File "C:\..\.virtualenv\Lib\site-packages\psygnal\_weak_callback.py", line 453, in cb
    func(obj, *self._args, *args, **self._kwargs)
  File "C:\..\.virtualenv\Lib\site-packages\pymmcore_plus\mda\handlers\_tensorstore_handler.py", line 204, in frameReady
    self._store = self.new_store(frame, event.sequence, meta).result()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psygnal.EmitLoopError:

While emitting signal 'pymmcore_plus.mda.events._psygnal.MDASignaler.frameReady', a ValueError occurred in a callback:

  Signal emitted at: C:\..\.virtualenv\Lib\site-packages\pymmcore_plus\mda\_thread_relay.py:144, in run
    >  emitter.emit(*args)

  Callback error at: C:\..\.virtualenv\Lib\site-packages\pymmcore_plus\mda\handlers\_tensorstore_handler.py:204, in frameReady
    >  self._store = self.new_store(frame, event.sequence, meta).result()

    Local variables:
       frame = array([[204, 197, 232, ..., 269, 224, 233],
       [214, 200...
       event = MDAEvent(index=mappingproxy({'t': 0, 'c': 0}), channel=Chann...
       meta = {'format': 'frame-dict', 'version': '1.0', 'runner_time_ms':...

See ValueError above for original traceback.

[tlambert03]

the problem in the log there is related to a pre-existing file that zarr doesn't want to overwrite. You can use the delete_existing parameter when instantiating the TensorStoreHandler() to avoid that. (Of course, micromanager-gui would need to do that for you when using the GUI directly)

I'm not sure it's directly related to the question of pausing between channels... could you perhaps also provide an example of the MDASequence you're trying to run, and perhaps the script you use to start the experiment (just in case there are any additional settings or details that might be relevant)

[simonecoppola]

It looks like the delete_existing was not the issue with the pausing, I just tried it. Here's the code I used:

from useq import MDAEvent, MDASequence
from pymmcore_plus.mda.handlers import TensorStoreHandler
mmcore = CMMCorePlus.instance()
mmcore.loadSystemConfiguration(path_to_config)
mmcore.setExposure(50)
mmcore.setCircularBufferMemoryFootprint(10000)

sequence = MDASequence(
    channels=[{"config": "Ex470-QUAD", "exposure": 50}, {"config": "Ex550-QUAD", "exposure": 50}],
    time_plan={"interval": 0, "loops": 100},
    axis_order="ct",
)
mmcore.run_mda(sequence, output=TensorStoreHandler(path=path_to_savefile, delete_existing=True,
                                                   driver="zarr"))

Here's the output, this time with no errors:

2024-09-30 11:10:05,819 - pymmcore-plus - INFO - (_runner.py:321) MDA Started: axis_order=('c', 't') channels=(Channel(config='Ex470-QUAD', exposure=50.0), Channel(config='Ex550-QUAD', exposure=50.0)) time_plan=TIntervalLoops(interval=datetime.timedelta(0), loops=100)
2024-09-30 11:10:05,820 - pymmcore-plus - INFO - (_runner.py:283) index={'t': 0, 'c': 0} channel=Channel(config='Ex470-QUAD') exposure=50.0 min_start_time=0.0 reset_event_timer=True events=(MDAEvent(index=mappingproxy({'t': 0, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0, reset_event_timer=True), MDAEvent(index=mappingproxy({'t': 1, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 2, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 3, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 4, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 5, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 6, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 7, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 8, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 9, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 10, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 11, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 12, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 13, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 14, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 15, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 16, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 17, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 18, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 19, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 20, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 21, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 22, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 23, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 24, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 25, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 26, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 27, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 28, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 29, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 30, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 31, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 32, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 33, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 34, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 35, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 36, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 37, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 38, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 39, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 40, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 41, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 42, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 43, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 44, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 45, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 46, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 47, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 48, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 49, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 50, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 51, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 52, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 53, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 54, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 55, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 56, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 57, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 58, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 59, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 60, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 61, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 62, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 63, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 64, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 65, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 66, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 67, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 68, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 69, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 70, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 71, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 72, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 73, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 74, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 75, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 76, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 77, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 78, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 79, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 80, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 81, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 82, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 83, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 84, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 85, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 86, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 87, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 88, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 89, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 90, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 91, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 92, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 93, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 94, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 95, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 96, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 97, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 98, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 99, 'c': 0}), channel=Channel(config='Ex470-QUAD'), exposure=50.0, min_start_time=0.0)) exposure_sequence=() x_sequence=() y_sequence=() z_sequence=()
2024-09-30 11:10:21,062 - pymmcore-plus - INFO - (_runner.py:283) index={'t': 0, 'c': 1} channel=Channel(config='Ex550-QUAD') exposure=50.0 min_start_time=0.0 reset_event_timer=True events=(MDAEvent(index=mappingproxy({'t': 0, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0, reset_event_timer=True), MDAEvent(index=mappingproxy({'t': 1, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 2, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 3, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 4, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 5, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 6, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 7, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 8, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 9, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 10, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 11, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 12, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 13, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 14, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 15, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 16, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 17, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 18, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 19, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 20, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 21, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 22, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 23, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 24, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 25, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 26, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 27, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 28, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 29, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 30, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 31, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 32, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 33, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 34, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 35, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 36, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 37, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 38, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 39, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 40, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 41, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 42, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 43, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 44, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 45, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 46, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 47, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 48, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 49, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 50, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 51, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 52, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 53, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 54, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 55, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 56, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 57, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 58, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 59, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 60, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 61, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 62, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 63, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 64, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 65, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 66, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 67, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 68, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 69, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 70, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 71, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 72, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 73, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 74, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 75, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 76, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 77, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 78, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 79, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 80, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 81, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 82, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 83, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 84, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 85, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 86, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 87, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 88, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 89, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 90, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 91, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 92, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 93, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 94, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 95, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 96, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 97, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 98, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0), MDAEvent(index=mappingproxy({'t': 99, 'c': 1}), channel=Channel(config='Ex550-QUAD'), exposure=50.0, min_start_time=0.0)) exposure_sequence=() x_sequence=() y_sequence=() z_sequence=()
2024-09-30 11:10:36,236 - pymmcore-plus - INFO - (_runner.py:408) MDA Finished: axis_order=('c', 't') channels=(Channel(config='Ex470-QUAD', exposure=50.0), Channel(config='Ex550-QUAD', exposure=50.0)) time_plan=TIntervalLoops(interval=datetime.timedelta(0), loops=100)

I should note that the behaviour is the same whether I run this just as a script, or with the micromanager-gui open. The only difference is that if the micromanager-gui is open, I see that it's catching up with the frames on the display. Thanks again for all your help with this!

[fdrgsp]

Hi @simonecoppola, I did a test on my microscope and it seems to works ok for me. Can you try too add the keep_shutter_open_across attribute to the MDASequence (this can be helpful if you have a shutter specified in the channels configs):

sequence = MDASequence(
    channels=[{"config": "Ex470-QUAD", "exposure": 50}, {"config": "Ex550-QUAD", "exposure": 50}],
    time_plan={"interval": 0, "loops": 100},
    axis_order="ct",
    keep_shutter_open_across="t",
)

I've also notice that in our MDAWidget we do not currently support the ct axis order (only tc) and maybe @tlambert03 we should add it (https://github.com/pymmcore-plus/pymmcore-widgets/blob/3dab0c4c22cc68916f51c2d9deccd0567d686c7a/src/pymmcore_widgets/useq_widgets/_mda_sequence.py#L51-L62)

[simonecoppola]

Hi @fdrgsp , I just tried adding the keep_shutter_open_across="t" but it made no difference :( it still acquired one channel, then paused for a bit, then did the other.

Regarding the support for ct, you totally should! It's useful for SRRF and not allowed in "normal" Micro-Manager (see https://forum.image.sc/t/scripting-complex-aquisitions-in-micro-manager/64555/7 and https://forum.image.sc/t/prearranging-complex-aquisitions-in-micro-manager/64992 for example of people being interested in the feature). It was actually what originally sent me towards pymmcore-plus, combined with needing to integrate some other stuff we already had written in python

[fdrgsp]

@simonecoppola can you paste here your cfg file? Thanks!

About the axis order, if you want to quickly enable it, I think you can comment out the ("t", "c"), # t cannot come after c line in your useq-schema code here https://github.com/pymmcore-plus/pymmcore-widgets/blob/3dab0c4c22cc68916f51c2d9deccd0567d686c7a/src/pymmcore_widgets/useq_widgets/_mda_sequence.py#L54.

[simonecoppola]

Sure thing, here it is: openframe_config.txt

[fdrgsp]

Hey @simonecoppola, is there any (do you hear any) hardware changing (e.g. shutter opening and closing, ...) while the acquisition is running and you see the "waiting" time between images? Do you have any delay set in any of the device you've added through the micromanager config wizard?

[simonecoppola]

Hi @fdrgsp, Nothing audible and no delays that I'm aware of!

I've done a bit more testing, and the delay between the two channels seems to be correlated with how many frames I'm acquiring (even if I'm just running a script and not the micromanager-gui), for example: if I acquire 10 frames for each channel, the delay is pretty minimal, if I increase this to 100 frames it gets longer etc. The behaviour does not change whether I am saving the data, or whether I'm just running mmcore.run_mda(sequence).

I've also run the same acquisitions while having the micromanager-gui open and not (i.e. just running a script), the delay between starting the acquisition for each channel is consistent regardless of the method, with the only difference being that when I have micromanager-gui open I can see that the acquisition for the second channel starts immediately after all the frames for the first channel have been displayed.

Thanks again for all your help with helping me troubleshoot this!

[fdrgsp]

Hey @simonecoppola, I am not sure what is happening...can you try to run this code and get info about the actual timing between frames (maybe start form the camera_metadata ElapsedTime-ms)?

import numpy as np
from pymmcore_plus import CMMCorePlus
from rich import print
from useq import MDAEvent, MDASequence

def on_frame_ready(img: np.ndarray, event: MDAEvent, metadata: dict) -> None:
    """Callback function to be called when a frame is ready."""
    print(metadata)

cfg = "/Users/fdrgsp/Desktop/test_config.cfg"
mmc = CMMCorePlus.instance()
mmc.loadSystemConfiguration(cfg)
mmc.mda.events.frameReady.connect(on_frame_ready)

sequence = MDASequence(
    channels=[{"config": "Ex470-QUAD", "exposure": 50}, {"config": "Ex550-QUAD", "exposure": 50}],
    time_plan={"interval": 0, "loops": 50},
    axis_order="ct",
    keep_shutter_open_across="t",
)

mmc.mda.run(sequence)

Another thing I would do is to try to use the demo camera instead of the Andor one and see if the "waiting" time is still there.

You could also try to remake the channels configs step by step by adding one device at a time, running the sequence, and then see when the timing issue occurs...