xpemon stuck in presence of a high rate of events

[albertomanfreda]

xpemon seems not to be able to deal with a rate of events grater than one every ~5 ms. It seems also to be consuming an unusual amount of memory. The two problems may be connected.

[lucabaldini]

200 Hz would be fine for most of the applications (although I am sure we can do more than that). The memory issue is more serious, and we should get to the bottom of it.

[albertomanfreda]

Ok I may have understood the issue with memory. The UDP socket receiving buffers and the class responsible for reading them (pEventReader) live on a different thread than the GUI and the histograms, using queued SLOT/SIGNAL connections to communicate. That is unavoidable, because we don't want the process of filling and drawing histograms to slow down the flow of input datagrams (also we need the GUI to stay operative during data reading). However, QT always makes deep copies of arguments passed through queued SLOT connections. That means, for example, that when pEventReader loops over a event and, for each pixel, emits a signal with its coordinates and the number of adc counts, those three variables (two doubles and an unsigned int) are copied at least one time (and maybe more) in the memory and stay there until all the SLOTs requesting them are executed. If the filling of the histograms is slower than the incoming flux of events, all these queued calls to SLOT waiting to be executed begin to flood the memory.

[albertomanfreda]

I have managed to improve a bit the performance (speed, memory usage) of the monitor.

• Speed: After a few tests I have discovered that the main bottleneck is the loop in the readDatagram() function of pEventReader. In this loop, a signal is emitted for each pixel to pass its coordinates and content to the other thread. I have changed the function so that only pixel above the zero suppression threshold are emitted. Of course that implies that the performance are now threshold-dependent. On my laptop the application is able to handle an event rate up to 500 Hz with a threshold of 5 or greater and to 1kHz with a threshold of 8 or greater, without any event loss.
• Memory: I have changed the way pEventReader works. The pEvent class is not used any more (and becomes candidate for deletion, though I suggest waiting a bit for that); instead, I directly call pDataBlock methods in readDatagram() to read data from the buffer . Doing so, we avoid an unnecessary loop on the buffer and an expensive copy. I have also made minor changes in the SIGNAL/SLOT logic, trying to minimize the amount of data passed through connections. However I can't really observe the effect of these changes with my test file of 500 events, so this requires testing with the detector.

[albertomanfreda]

I have realized that is much more efficient to fill the histograms in the same thread where the UDP socket operates. A copy of the histograms (which is used for plotting) is kept in the GUI thread and it is periodically synchronized with the other one. This scheme minimize the quantity of inter-thread communication.

A test with 500 events on my laptop (Ubuntu 14.04) and on one PC in the lab (Windows) has shown no slowing of the application and no data loss up to 1 kHz (and I definitely think we are able to handle an even higher rate). This result is threshold independent.

I am closing the issue.