Add support for sequence acquisitions with external triggering

[henrypinkard]

Pycro-manager by default supports hardware triggered acquisitions where the camera is the master device. An alternative to this are situations where where there is another master device and the camera waits to receive triggers from it. Implementing this situation will require a new acquisition hook that runs after the camera has been told to start waiting for triggered acquisitions. This hook will be used to send a custom command to whatever devices is the master of the triggering setup.

[dpshepherd]

For the specific use case that we are hoping to solve right now, there is only one external trigger for each sequence. The external trigger starts the camera, which then generates images at the set frame rate until it receives a software stop signal. The frame rate, ROI, etc... are all configured using software calls before the camera is set to the 'Trigger First' mode.

The more general case you have described will also be useful, but I'm not sure if there will be a practical difference in supporting these cases.

[henrypinkard]

No it should be the same since the hook will only run once for each sequence

[henrypinkard]

@dpshepherd I added support for this that I think should work. Want to give it a test? It should be available in the next nightly build of MM after https://github.com/micro-manager/micro-manager/pull/893 gets merged. New features are available in pycromanager 0.6.0 (which is now on pypi).

You should be able to modify this example to make it work. Let me know if you have questions.

[dpshepherd]

Awesome! Should be able to get to this next week and provide some feedback.

[dpshepherd]

Two questions came up about implementing this for our setup based on your example.

1. We don't want to setup anything in the acquisition except the number of images to capture. Would we just set num_time_points equal to the number of images expected, keep time_interval_s=0, and delete the rest of the setup? The external controller will handle the rest.
2. Is there a reason a we should not pass additional parameters img_process_fn using a functools.partial object? We would like to move away parsing/saving the MM metadata structure because it really bogs down when we capture millions of images.

[henrypinkard]

1. Yes, you could do this as just one big timelapse. Though if you plan to use the built in viewer it might be better to create custom acquisition events (rather than using the multi_d_acqusition convenience function) and supply z indices, but nothing related to the hardware.

2. I think that should work fine. I've set attributes on img_process_fn in order to save some kind of state from call to call, which I suspect might be using the same internal mechanism as partial (though I've not used the latter myself)

if you are trying to save to your own format in real time, you will be limited by the speed with which the transfer layer can send data from Java to Python. I think this is around 100 MB/s. The native Java saving can go a lot faster than this if you have the right hardware and don't use an image processor. Also, this saving uses a different storage library than the regular MM one. Though they are similar, this one is optimized for speed and scale. So you may not run into the same problems as before.

[dpshepherd]

Ah, I didn't realize the max data transfer rate of the bridge. We are well in excess of that (order of magnitude). We will try it out but will probably have to find another path forward for the particular scope I had in mind.

We have another rig where we don't generate data as rapidly that this should be useful for.

[henrypinkard]

It's possible if your RAM is big enough to hold all or most of a full acquisition you could cache the images as they are sent across the bridge.

Another better possibility would be that you can just read the data as soon as the acquisition ends and resave in a different format. Could be in the same script so you wouldnt need to run a separate step.

Out of curiosity, what is it about the tiff-based format that doesn't work for your analysis?

[dpshepherd]

Holding into RAM isn't an option, because we are acquiring at minimum > 7 TB of raw imaging data and need to stream it to disk as fast as possible.

We are acquiring tilted planes with a high NA stage scanning oblique plane microscope. We have to deskew these somehow. This can be done by directly writing to a bigdataviewer H5 and baking in the needed transforms OR do orthogonal interpolation once a given "block" (typically one laser with millions of images acquired at ~200 nm spacing during a constant speed stage scan) is done. It is much more efficient to save as the H5 format as a first step for these massive datasets.

We are already doing your second suggestion with our micromanager 2.0 based acquisition. We use a separate Python program that polls the directory structure and starts the deskewing / resaving the TIFF file as soon as one "block" has finished acquiring.

[henrypinkard]

Makes sense. I tend to think in general that it is better to not do any intensive processing on the fly if you dont have to and you're really trying to push data throughput, and it's not clear that writing directly to hdf would be able to keep up given the performance issues with h5py.

There are other things you could try, such as writing a Java-based image processor to avoid going through the bridge, but there are not guarantees these would work at the performance level you need. Might not be worth the trouble compared to the current strategy

[dpshepherd]

Yes, I agree. We had some other reasons (fluidics control) that we were interested in pycro-manager.

There are performant ways to write from numpy to h5 within Python. For example, https://github.com/nvladimus/npy2bdv. One can also set up a custom solution and compress on the fly using something like B3D https://github.com/balintbalazs/B3D. We have a pure Python based control code for this, but our current camera's Python SDK is a bit annoying and has nice MM drivers.

On Mon, Aug 3, 2020 at 11:29 AM Henry Pinkard notifications@github.com wrote:

Makes sense. I tend to think in general that it is better to not do any intensive processing on the fly if you dont have to and you're really trying to push data throughput, and it's not clear that writing directly to hdf would be able to keep up given the performance issues with h5py https://cyrille.rossant.net/moving-away-hdf5/.

There are other things you could try, such as writing a Java-based image processor to avoid going through the bridge, but there are not guarantees these would work at the performance level you need. Might not be worth the trouble compared to the current strategy

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/micro-manager/pycro-manager/issues/60#issuecomment-668175361, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGMK4VVXGT4V6UF6DHBZ2A3R636Z7ANCNFSM4OHLVC6Q .

[henrypinkard]

Good to know. Maybe in the future a better transfer layer can make this possible, but it's not on my immediate todo list

Let me know if the triggering stuff works as expected