search

mfrey / baltimore

baltimore is an analysis script for the libARA framework
GNU General Public License v3.0
0 stars 0 forks source link

Add multithreading support #12

Closed mfrey closed 11 years ago

mfrey commented 11 years ago

We should support multithreading (by means of the multprocessing package)

mfrey commented 11 years ago

I've adde simple multiprocessing support for baltimore. For each scenario a process is generated which handles the parsing of results and the analysis of the data. There is currently no further process control implemented which might be required at a later point for time (for example for putting results across scenarios 'together' for graphical output (by means of boxplots ...)). I think it is safe to close this issue and open a new one for further requirements.

fgrosse commented 11 years ago

Oh I thought the multithreading support would be aimed at speeding up the analysis of single scenarios as those are to be evaluated much more often. I can open another issue for multithreaded reads (parsing) of the result files or should we reopen this one?

mfrey commented 11 years ago

We can still speed this up by 'multiprocessing' the repetitions (meaning one process per repetition). I know that multithreading seems to be the more intuitive approach here (sharing address space and minimizing the overhead of processes), but due to the global interpreter lock multithreading in Python does not perform very well and should not be used. Right now (and it's not commited) I start for every scenario in the results directory a process and the result is stored in a queue (meaning that this is the producer) which is owned by class ExperimentManager (the consumer). I'm going to add the repetitions as processes as well (but this might take some time).