search

eqcorrscan / RT_EQcorrscan

Real-Time implementation of EQcorrscan methods.
https://rt-eqcorrscan.readthedocs.io/
GNU General Public License v3.0
13 stars 6 forks source link

Move Streamers from Threads to Processes #13

Closed calum-chamberlain closed 3 years ago

calum-chamberlain commented 3 years ago

I ran into some issues when running multiple RealTimeTribes with many templates that data would appear gappy from seedlink clients. I suspect that this was due to the MainProcess hogging the thread-time and not allowing the Streamer any of the Thread.

To get around this I have migrated the Streamers to Processes, with associated Queues for communication. This has been quite complicated!

I'm running a longer-term test with the same settings that resulted in the issue to see if this crops up again...

calum-chamberlain commented 3 years ago

At the moment multiprocessing is not working on Windows or MacOS, I think because the Streamer classes are not pickleable, and Windows uses a different multiprocessing process instantiation. For now I have set it so that on Windows threading is used, and everywhere else, multiprocessing is used.

This may mean that Windows and MacOS users will run in to rate-limitations and gaps in data. If anyone on windows finds this PR in the future, this might be why!