ETA Streaming

[linzuzeng]

Currently ETA is hard-coded to read timetags from a file. Sometimes the software from acquisiton devices behaves strangely and the timetag file might not be good for read during acquisition. However, almost all acquisiton devices provide their API that could read timetags directly into a pre-allocated memory buffer. We can implement the ETA Streaming mode that directly reads the timetags from that buffer.

We could add a "memory source" as an alternative to the existing "file source". Users can implement the loading of their own format and put timetags into an array (also known as the Swabian Instrument format), and pass it directly into eta.simple_cut()

[linzuzeng]

quTag proviedes TDC_getLastTimestamps and TDC_setTimestampBufferSize, maybe we could use them

[linzuzeng]

We can either make a thread to copy the content from the buffer to the file, and run the old ETA realtime against that file

Or we can add a new mode to the cut functions that takes an address of the buffer instead of the file

In the second we we can also save the file, but we avoid one unnecessary read

and if the countrate is really high, we can avoid writting to disk in the second way

[linzuzeng]

I measured the performance loss of several possible ways to migrate the file reading from C to Python.

It turns out that the file.read_into() method reads almost as fast as fread() in C (less than 1% performance loss), and it clearly out-performs other methods like np.fromfile() and Pathlib.read_bytes(), which are already considered to be very fast file IO methods in the Python but actually almost doubled the fread() reading time when performing reading in 10MB chunks.

The overhead caused by switching between Python and native assembly is about 1% when reading the file in 10MB chunks. But it shouldn't be a major issue since the switching time is constant and an increased chunk size would lead to less switching and make it more neglectable.

[linzuzeng]

I will try to implement a unified buffer manager which manages all timetag buffers for both read and write, which will make #104 easier.

The unified buffer will be allocated, filled with timetag file or realtime timetag sources, and recycled in Python, thus we can remove all platform-related API from the existing C code, which will also make ETA work on the Mac OS.

Since the buffer is read-only when doing ETA analysis, we can also reuse the memory address or byte array filled by the timetagger library, realizing in-memory and Zero-copy analysis, which would make ETA realtime faster :)

[linzuzeng]

Supported. Reopen when we find new problems.