Refactored logging so that child processes can still write to the stdout stream.
The basic gist is this:
-- we make a multiprocesses Queue that listens for log messages and outputs them to the stdout:
# in the main process, eg `acquire_video()`
logger_queue = mp.Queue()
queue_listener = QueueListener(logger_queue, StreamHandler())
queue_listener.start()
Then we make some child processes to which we pass the same Queue, and we set up a logger in the run method of the child process to use that Queue:
# in run() of a child proc
logger = logging.getLogger(process_name)
# Add a handler that uses the shared queue
handler = logging.handlers.QueueHandler(logger_queue)
logger.addHandler(handler)
For some unknown reason, PyTest doesn't automatically display the logs from the child processes the way it does the logs from the main process. However, printing the logs with pytest ... -s or running the tests as a direct python script work just fine, so I'm happy for now.
Refactored logging so that child processes can still write to the stdout stream.
The basic gist is this: -- we make a multiprocesses
Queue
that listens for log messages and outputs them to the stdout:Then we make some child processes to which we pass the same
Queue
, and we set up a logger in therun
method of the child process to use thatQueue
:For some unknown reason, PyTest doesn't automatically display the logs from the child processes the way it does the logs from the main process. However, printing the logs with
pytest ... -s
or running the tests as a direct python script work just fine, so I'm happy for now.