We define tasks as subclasses of multiprocessing.Worker and use it's run method to execute tasks in parallel. However these have no communication with the main thread and thus no way of retrieving the data it generates.
This means that tasks that run with this lib must themselves handle the output, currently to the filesystem, them later aggregate them in a single file.
By implementing some sort of inter-process communication in libparallel we can drop this i/o operations and potentially even reduce the number of files - as a similar logic can be applied to libsubprocess
We define tasks as subclasses of
multiprocessing.Workerand use it'srunmethod to execute tasks in parallel. However these have no communication with the main thread and thus no way of retrieving the data it generates.This means that tasks that run with this lib must themselves handle the output, currently to the filesystem, them later aggregate them in a single file.
By implementing some sort of inter-process communication in
libparallelwe can drop this i/o operations and potentially even reduce the number of files - as a similar logic can be applied tolibsubprocess