This PR changes detectors run by a Reactor to run as subprocesses rather than using multiprocessing. No communication is required between the Reactor and the individual detectors, with the Reactor controlling which templates are run by which detector.
New templates are added by writing a new tribe file into the individual detector working directory. The detector checks for this file and reads from it and removes it once it is done. In this way templates can be added to a running detector, while it is running.
As an added advantage, this allows subprocessed detectors to write to their own log-file (which is what I wanted). This keeps logs seperate making it easier to understand what process is doing what. This closes #5
The main reason for doing this was that memory would increase monotonically with the number of detectors running. This was due to the use of os.fork under the hood when spawning new processes. I tried using os.spawn, but ran into some other issues that I forget. os.fork replicates the memory for each new process. This was not ideal.
This PR changes detectors run by a
Reactorto run as subprocesses rather than using multiprocessing. No communication is required between theReactorand the individual detectors, with theReactorcontrolling which templates are run by which detector.New templates are added by writing a new tribe file into the individual detector working directory. The detector checks for this file and reads from it and removes it once it is done. In this way templates can be added to a running detector, while it is running.
As an added advantage, this allows subprocessed detectors to write to their own log-file (which is what I wanted). This keeps logs seperate making it easier to understand what process is doing what. This closes #5
The main reason for doing this was that memory would increase monotonically with the number of detectors running. This was due to the use of
os.forkunder the hood when spawning new processes. I tried usingos.spawn, but ran into some other issues that I forget.os.forkreplicates the memory for each new process. This was not ideal.