Acquisition Events as Generator

[carlkesselman]

I am doing some long duration acquisitions with a lot of events. I am using a generator to create these event sequences. However, the validation code in the acquisition module explicitly is looking for a list, and also iterates through that list up front to check each event.

I'm curious as to if providing an alternative mode where a generator could be provided as an alternative. In this case, you would probably loose the ability to validate the entire acquisition up front, but would have the advantage of being able to provide a large number of events with a small memory footprint.

I would be happy to code this up and do a PR if this seems like a good idea.

Thanks,

Carl

[henrypinkard]

Could you just make multiple calls to acquire, like this:

with Acquisition(...) as acq:
   for event in your_generator():
      acq.acquire(event)

You would not be able to do hardware sequencing over multiple calls to acquire, but perhaps this doesn't matter in your application? I'm curious to know if this works for you

[carlkesselman]

Hmmm… I wonder.

I am using an external trigger for this, but my frame rates are pretty slow. So if I were do to this one event at a time, how would the interface know to wait for the external trigger. Or would this just work because the camera is configured to wait for an external trigger?

In this case, I wouldn’t need the post_camera_hook, right?

My use case is that I’m generating two hours of acquisition at 30FPS.

Carl

On Mar 27, 2023, at 2:38 PM, Henry Pinkard @.***> wrote:

Could you just make multiple calls to acquire, like this:

with Acquisition(...) as acq: for event in your_generator(): acq.acquire(event)

You would not be able to do hardware sequencing over multiple calls to acquire, but perhaps this doesn't matter in your application? I'm curious to know if this works for you

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v2/url?u=https-3A__github.com_micro-2Dmanager_pycro-2Dmanager_issues_571-23issuecomment-2D1485894325&d=DwMCaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=NEvDV_KflySJQQ3LGzJYlioj--G7cw9WYsc2xbn4o8g&s=2ICHEOxoGmEHR3BxUHtsLKlWNfqzEf61jmiFKxyOQQc&e=, or unsubscribehttps://urldefense.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AA3OGXWQ35E4PNKKUMSZUF3W6ICELANCNFSM6AAAAAAWJKYD5M&d=DwMCaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=NEvDV_KflySJQQ3LGzJYlioj--G7cw9WYsc2xbn4o8g&s=2zM7IF3gHvXd49MSRQt4w4BH64B15MSzlDf63K_H31w&e=. You are receiving this because you authored the thread.Message ID: @.***>

[carlkesselman]

Actually, I would have to restructure my application a bit, as I’m collecting data from multiple cameras using headless mode.

Carl

On Mar 27, 2023, at 4:59 PM, Carl Kesselman @.***> wrote:

Hmmm… I wonder.

I am using an external trigger for this, but my frame rates are pretty slow. So if I were do to this one event at a time, how would the interface know to wait for the external trigger. Or would this just work because the camera is configured to wait for an external trigger?

In this case, I wouldn’t need the post_camera_hook, right?

My use case is that I’m generating two hours of acquisition at 30FPS.

Carl

On Mar 27, 2023, at 2:38 PM, Henry Pinkard @.***> wrote:

Could you just make multiple calls to acquire, like this:

with Acquisition(...) as acq: for event in your_generator(): acq.acquire(event)

You would not be able to do hardware sequencing over multiple calls to acquire, but perhaps this doesn't matter in your application? I'm curious to know if this works for you

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v2/url?u=https-3A__github.com_micro-2Dmanager_pycro-2Dmanager_issues_571-23issuecomment-2D1485894325&d=DwMCaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=NEvDV_KflySJQQ3LGzJYlioj--G7cw9WYsc2xbn4o8g&s=2ICHEOxoGmEHR3BxUHtsLKlWNfqzEf61jmiFKxyOQQc&e=, or unsubscribehttps://urldefense.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AA3OGXWQ35E4PNKKUMSZUF3W6ICELANCNFSM6AAAAAAWJKYD5M&d=DwMCaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=NEvDV_KflySJQQ3LGzJYlioj--G7cw9WYsc2xbn4o8g&s=2zM7IF3gHvXd49MSRQt4w4BH64B15MSzlDf63K_H31w&e=. You are receiving this because you authored the thread.Message ID: @.***>

[henrypinkard]

I think this is a good use case for expanding the capabilities of the acquisition engine and to have some kind of event that means "acquire continuously until a certain time has elapsed", essentially signaling the creation of a generator as you suggest. It would be a good addition. This will also become easier/more powerful with the addition of a new camera API (https://github.com/micro-manager/mmCoreAndDevices/issues/243)

(Linking https://github.com/micro-manager/micro-manager/issues/1524 because that's where discussion of acquisition engine improvements will occur)

However, I think you maybe can currently get by by calling acquire one event at a time. Each event will lead to a call of the post_camera_hook, and you'll have to constantly send trigger pulses for your camera. I'm guessing there would be too much overhead to do this at a high framerate, but maybe will be okay in your case? What do you think?

[carlkesselman]

Hi,

I”m trying to get acquire to work one event at a time, but I’m seeing inconsistent behavior with respect to running the post_camera_hook_fn. Everything is ok on my uEye cameras, but the hook never gets called on my hammamatsu camera.

What is the expected behavior of this hook with respect to individual events, Here is the code I’m using:

for e in self._events: self.acquisition.acquire(serialize(e)) self.acquisition.mark_finished()

I get a different behavior than:

self.acquisition.acquire([serialize(e) for e in events])

THanks,

Carl

---

Dr. Carl Kesselman William H. Keck Profesor of Engineering Professor, Epstein Department of Industrial and Systems Engineering Fellow, Information Sciences Institute Viterbi School of Engineering

Professor, Department of Population and Public Health Sciences, Keck School of Medicine Ostrow School of Dentistry

University of Southern California 4676 Admiralty Way, Suite 1001, Marina del Rey, CA 90292-6695 Phone: +1 (310) 448-9338 Email: @.*** Web: http://www.isi.edu/~carl

On Mar 28, 2023, at 9:47 AM, Henry Pinkard @.***> wrote:

I think this is a good use case for expanding the capabilities of the acquisition engine and to have some kind of event that means "acquire continuously until a certain time has elapsed", essentially signaling the creation of a generator as you suggest. It would be a good addition. This will also become easier/more powerful with the addition of a new camera API (micro-manager/mmCoreAndDevices#243https://urldefense.com/v2/url?u=https-3A__github.com_micro-2Dmanager_mmCoreAndDevices_issues_243&d=DwMCaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=uo_RtgeIXlElSfmgrewWLq2tu1OTV8qZuTS0-rzExac&s=WaLNLcwI-Uc6OHYa_sVVa0iyLqvh7FAgLOBsSClXQRM&e=)

(Linking micro-manager/micro-manager#1524https://urldefense.com/v2/url?u=https-3A__github.com_micro-2Dmanager_micro-2Dmanager_issues_1524&d=DwMCaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=uo_RtgeIXlElSfmgrewWLq2tu1OTV8qZuTS0-rzExac&s=dRUGl594hNuOqblEOPJ0MWpS0yYc0Tm_D702Rre7aMY&e= because that's where discussion of acquisition engine improvements will occur)

However, I think you maybe can currently get by by calling acquire one event at a time. Each event will lead to a call of the post_camera_hook, and you'll have to constantly send trigger pulses for your camera. I'm guessing there would be too much overhead to do this at a high framerate, but maybe will be okay in your case? What do you think?

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v2/url?u=https-3A__github.com_micro-2Dmanager_pycro-2Dmanager_issues_571-23issuecomment-2D1487271566&d=DwMCaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=uo_RtgeIXlElSfmgrewWLq2tu1OTV8qZuTS0-rzExac&s=7sDoog1TdvP15aOvnUH4cHr_C5V-bZshhlO8TFHoOMI&e=, or unsubscribehttps://urldefense.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AA3OGXXMXZR7LVUCJQ77CRTW6MIZVANCNFSM6AAAAAAWJKYD5M&d=DwMCaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=uo_RtgeIXlElSfmgrewWLq2tu1OTV8qZuTS0-rzExac&s=9H6iPOKuJ2KUy1XR0UiCzvNZpexrbb1_yIWrGg2u4io&e=. You are receiving this because you authored the thread.Message ID: @.***>

[henrypinkard]

Why are you calling serialize?

[carlkesselman]

That is my routine to conferred a data class that represents an event into a dictionary with no None element. Sorry. Confusing.

Carl

Sent from my iPhone

On Mar 28, 2023, at 3:55 PM, Henry Pinkard @.***> wrote:



Why are you calling serialize?

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v2/url?u=https-3A__github.com_micro-2Dmanager_pycro-2Dmanager_issues_571-23issuecomment-2D1487706980&d=DwMCaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=BgwAIkFXuPXUN5WI1013mkoq7IIO6Ts2q5lo0QxZ1g8&s=YNrpQORncg3eqEDIeuFkrVEP8-ip2zr97b7zc8umr9A&e=, or unsubscribehttps://urldefense.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AA3OGXQQJ5QFETV5PMHWQ6LW6NT5XANCNFSM6AAAAAAWJKYD5M&d=DwMCaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=BgwAIkFXuPXUN5WI1013mkoq7IIO6Ts2q5lo0QxZ1g8&s=LMnF8lEHUGyTSbepbhT9kgWmLGXSM_9mnPWKm1ssvu8&e=. You are receiving this because you authored the thread.Message ID: @.***>

[henrypinkard]

I'm a bit confused now. I recommend that you make a hook function that simply prints the events and passes the event along, because this may clarify what is going on. Also, trying it out on the demo camera might be a good idea to control for the variable of different camera types

[ieivanov]

@carlkesselman do you have the option to dispatch events in hardware-sequenced batches? It may be the case that your entire 2h 30 fps acquisition is not sequenced. Instead, for example, z stacks for a given channel within this acquisition are sequences. In that case you would call acquire multiple times with sets of events corresponding to the hardware-sequenced z stack portion of the acquisition.

[carlkesselman]

Thanks…. that is exactly what I ended up doing. I’m chunking the acquisitions and dispatching them with some overlap so there is no gap. I’m fighting with my cameras right now but I think this is working. My situation if further complicated in that I’m running three cameras by using the headless_start feature.

I think there is an interesting underlying question as to how big an acquisition sequence is considered to be reasonable. We have experiments that run for hours, and we are acquiring some data at 30FPS.

Thanks,

Carl

On Apr 3, 2023, at 4:55 PM, Ivan Ivanov @.***> wrote:

@carlkesselmanhttps://urldefense.com/v2/url?u=https-3A__github.com_carlkesselman&d=DwMCaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=4HSOumH-9jXR9vdwIsR7bL4S6nOfF3W73XgVvculW6I&s=XAUtYs7BM7Zjz-Z4RPI6qljPBs5FO7xtQApkCLKvNp0&e= do you have the option to dispatch events in hardware-sequenced batches? It may be the case that your entire 2h 30 fps acquisition is not sequenced. Instead, for example, z stacks for a given channel within this acquisition are sequences. In that case you would call acquire multiple times with sets of events corresponding to the hardware-sequenced z stack portion of the acquisition.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v2/url?u=https-3A__github.com_micro-2Dmanager_pycro-2Dmanager_issues_571-23issuecomment-2D1495139709&d=DwMCaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=4HSOumH-9jXR9vdwIsR7bL4S6nOfF3W73XgVvculW6I&s=qcB0R_gXxwbYFmF0WdnvAZap6Bd0cezb8N0FPyFuH4k&e=, or unsubscribehttps://urldefense.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AA3OGXXF2N23LOYEFAVC4HDW7NPPHANCNFSM6AAAAAAWJKYD5M&d=DwMCaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=sGCma2ufaUVT-N141kRIZQ&m=4HSOumH-9jXR9vdwIsR7bL4S6nOfF3W73XgVvculW6I&s=coxnEl6fsfq8KdKeNDPrRq65zh6CNFUvI7ctTxMe68o&e=. You are receiving this because you were mentioned.Message ID: @.***>