best practices in writing an MDA engine?

[ianhi]

In https://github.com/micro-manager/mmCoreAndDevices/issues/171#issuecomment-1058312997 Mark said:

MDA, MMStudio (via acqEngine) pops every image that gets placed in the sequence buffer.

which surprised me because I would not have thought the sequenceAcquisitiion would be easy to coordinate with software based updates to hardware like stages and it would be easier to update the hardware and then call snap (what pymmcore-plus currently does). But then I poked around poked around in acqEngine for a bit and think I understand what's going on there. So i have two questions:

1. Is the below understanding of what acqEngine is doing correct?
2. What's the best practice in writing an MDA engine? Are there things that could be improved on in acqEngine or should we look to that as the gold standard?
   • In pymmcore-plus we provide a default mda engine
   • We also recently made it easy for users to define their own acquisition engines. It would be nice to be able to provide advice about this to people make their own engines that incorporate custom hardware or have stricter performance requirements.

---

My understanding of what acqEngine is doing is that it has the option to use either snap or sequence:

https://github.com/micro-manager/micro-manager/blob/47298d8df3a3c8013a900e5d7274a8c7e46948a7/acqEngine/src/main/clj/org/micromanager/acq_engine.clj#L581-L592

it prefers to use a sequence if that's an option for the given hardware, which is determined by this code:

https://github.com/micro-manager/micro-manager/blob/47298d8df3a3c8013a900e5d7274a8c7e46948a7/acqEngine/src/main/clj/org/micromanager/sequence_generator.clj#L221-L237

[marktsuchida]

• Is the below understanding of what acqEngine is doing correct?

Yes. Snaps are the default, but sequences are used automagically when allowed/possible. The basic idea is that acqEngine (lazily) generates a fully written out sequence of "snaps", then coalesces the first however many that are compatible with each other into a sequence. Compatibility is determined by the property and stage position differences and other factors (such as the need to perform autofocus, lack of specific time interval, and maximum sequence length supported by devices). There is an (incomplete) description for users.

You can see the effect of this design by observing that a simple time series (0 time interval) takes a long time to start up when the frame count is large (due to the explicit generation of the snap sequence before coalescence).

• What's the best practice in writing an MDA engine? Are there things that could be improved on in acqEngine or should we look to that as the gold standard?

I can't answer this question fully here; it would take many days, and I have many thoughts and ideas that I've accumulated over the years. What I can say for sure is that this is not a best practices question, but rather a serious software design challenge.

It is clear to me that there are many problems with the design of acqEngine. On the other hand, I have to say that acqEngine achieves quite impressive things on a good day, relative to its simple(?) design -- enough to make it hard to replicate all of its features in a new design.

Some of the problems that come to mind, specific to hardware sequencing:

• User cannot tell if hardware sequencing will happen until running the acquisition (a troubleshooting nightmare)
• Enabling or disabling sequencing (for particular axes) is rather indirect (and relies on copy-pasted code in device adapters)
• The lazy sequence generation is very inefficient where long sequences are possible (as I mentioned above)
• Some of the rules for breaking up sequences when they are too long do not always result in desirable behavior
• The T-P-Z-C axes and autofocus behavior are hard-coded (a problem not specific to sequencing) (but there is definitely a tradeoff here between overly abstract generalization and intuitive simplicity)
• In general, the mechanism is very opaque and only observable through the CoreLog (and, realistically, by an expert).

One possible alternative design would be to separate the "compilation" of an MDA definition into hardware operations (snaps, acquisition and property sequences, and other non-sequence operations) from the "execution" of the latter. That way, we'd know exactly what is going to happen before starting the acquisition. It also potentially facilitates expressing various bells and whistles in a more modular fashion. But this implies coming up with a good intermediate representation (IR) that contains loops as first-class objects, and probably defining a series of transformations ("optimization passes") upon such an IR. The compiler analogy is actually quite interesting, because something similar to loop unrolling might occur.

I spent some time thinking about such a design back in 2014 or so, and it seemed clear that a nontrivial amount of effort would be needed. It is hard to say if it's the best way to go -- having a solid software-sequenced engine with a few strategic special cases and a good set of hooks (extension points) might support 90% of the use cases with 10% of the effort, and may not be that bad if the special cases are well isolated.

There are many more topics. Autofocus cycles really complicate the current design, but are also not that easy to just tack on to an otherwise simplistic design. The desire to run software-sequenced motion of multiple devices (stages, turrets) concurrently is another complication, because often you need temporal overlap between adjacent iterations of a loop (the diff-based method of acqEngine is one elegant solution to this specific aspect). There is no reason why the features in MM's MDA dialog should be considered the baseline requirements (for example, rapid time series repeated at long intervals is a common demand that is not satisfied). For users who can script, it might be much more beneficial to have some reusable tools to simplify writing a custom acquisition loop rather than having to hook into an overly opaque "engine" -- but a well-designed and customizable engine might be even better.

The one thing that feels clear is that the challenge is in coming up with a good way to factor the different concerns -- one that is not merely conceptually clean, but can actually support a wide range of practical considerations and use cases.

[ianhi]

Thank you mark! I need some time to process all of that, but one comment jumped out at me:

It also potentially facilitates expressing various bells and whistles in a more modular fashion. But this implies coming up with a good intermediate representation (IR) that contains loops as first-class objects, and probably defining a series of transformations ("optimization passes") upon such an IR.

We talked a bit about using a state machine schema to define expreiments over on useq-schema. Would be really curious to have your input there: https://github.com/tlambert03/useq-schema/discussions/28 and https://github.com/tlambert03/useq-schema/discussions/30

[marktsuchida]

Interesting. I'm not sure I'm grasping exactly how you intend to use state machines from those comments (what are the states and transitions going to represent?). So instead of commenting there, I'll just mention some random thoughts here.

At some level, the fact that acquisition sequences can be modeled as state machines is self-evident. But whether state machines are the most useful model for thinking about acquisitions, or for composing inner and outer loops, or for visualizing toward a user, or for providing an approachable and comprehensible representation, is probably heavily dependent on the details of how they are constructed and organized. This is kind of like how people think differently in different programming languages, even if they are all "equivalent" (in the Turing completeness sense). So I feel a general comment is hard to make.

I have to say I'm not familiar with all the features that some of these newer state machine frameworks/representations provide (I've been keeping an open mind and expecting there is a lot of good stuff that is not evident in the toy examples that are invariably shown, but I haven't had a chance to dive deeper), so I don't have a good feel for how easy it is to deal with concurrent tasks (I'm sure they've thought about this if they are considered useful tools for cloud operations), and for hierarchically (or otherwise) composing state machines. I do think that they are at the very least one good candidate for the transformable representation I hinted at above.

One thing I worry about is whether using a state machine framework/library/language/schema could lead to excessive abstraction -- I'm thinking of how simple concepts can become really hard to grasp when put into, say, XSLT (or even just XML, depending on how complex the schema is). In some sense, general state machines might be too powerful a representation. Whether this is important may depend somewhat on whether the audience for the format is just the core developers or advanced users or all users. Maybe it's not a problem if a bijection exists to a good graphical/interactive representation.

Also, as simple as the concept of state machines might be, my impression is that it takes a certain kind of mindset to construct them correctly. While it is much easier to explain to a non-programmer what a state machine is than to explain how async/await work, my gut feeling is that it is easier to teach somebody how to use async/await than it is to teach them to actually construct correct and efficient state machines for a real-life problem, because in the former case, the necessary states arise automatically (if implicitly), so to speak. This, again, probably depends a lot on exactly how the concepts are used and presented.

I'm not sure how helpful these abstract thoughts are -- I might have more to say given more details of what you have in mind.