Triggering of xspress3 at BMM takes 2.5s for 0.5s exposure

NSLS-II / nslsii

NSLS-II related devices

BSD 3-Clause "New" or "Revised" License

10 stars 21 forks source link

Triggering of xspress3 at BMM takes 2.5s for 0.5s exposure #96

Open mrakitin opened 3 years ago

mrakitin commented 3 years ago

This part of the code seems to be a bottleneck (see the attached picture with the ts_msg_hook-generated output): https://github.com/NSLS-II/nslsii/blob/18c06329edb76f068d63e5b6522a6781d2bc696c/nslsii/detectors/xspress3.py#L623-L638

20200813_155248

mrakitin commented 3 years ago

Did a few tests today with @bruceravel, all repeatable, even with the disabled HDF5 plugin.

@tacaswell, what would be the best way to debug it? Which kinds of prints can help here? I guess this dispatching is doing something very slowly, but not sure how to best approach it.

jklynch commented 3 years ago

https://github.com/joerick/pyinstrument

tacaswell commented 3 years ago

I think we may need to throw a whole lot of camonitors on the AD signals to see where it is getting stuck. If we are at the wait message, than that means we are through trigger and are waiting now waiting for the code in https://github.com/NSLS-II/nslsii/blob/18c06329edb76f068d63e5b6522a6781d2bc696c/nslsii/detectors/xspress3.py#L615-L621 to fire.