Open dongbohu opened 4 years ago
This article discusses logging in multiprocessing
module: https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes
It seems to make sense that logging in child processes is independent from logging of parent process. But why does this only happen on macOS?
This issue is also confirmed on macOS 10.13.6 (High Sierra) running Python 3.8.1.
On a macOS test box (Mojave, 10.14.6, running Python 3.8.1), when parallel data processing is enabled by
-parallel n
option (in whichn
> 0), most of the log messages generated by child processes disappeared from the main console.For example, when running the test case in
examples/gwas/gwas.yml
, here is the output generated bycimr processor -process -yaml-file gwas.yml -chunksize 500 -parallel 4
:Only five lines of log messages from child processes were shown above:
But if we change the command to non-parallel mode:
cimr processor -process -yaml-file gwas.yml -chunksize 500 -parallel 0
then the output log messages became:All lines from:
until:
were generated by child processes.
This issue doesn't happen on Linux boxes (Ubuntu and Debian, Python 3.6.x, 3,7,x and 3.8.1).