BrokenProcessPool encountered when passing logging configuration to spawn subprocess on linux (hashtag1160)

[vikramsubramanian]

I am attempting to get loguru working for spawned subprocesses and have been running into a concurrent.futures.process.BrokenProcessPool crash after attempting to follow the advice in issues such as and as illustrated in this PR:

This crash only happens on Linux, the issue is not reproducible on macOS.

I am hoping for some guidance on what might be wrong here and if this is something that could be fixed within loguru, or that I can change on my end.

This is a program that reproduces the issue:

import multiprocessing
import sys
from concurrent.futures import ProcessPoolExecutor, as_completed

from loguru import logger

def _processpool_initializer(parent_logger):
    logger._core = parent_logger._core  hashtag type: ignore

    print("before failure", flush=True)
    logger.info("BOOM")
    print("after failure", flush=True)

def upper(v):
    return str(v).upper()

def main():
    logger.remove()
    logger.add(sys.stdout, enqueue=True)

    executor = ProcessPoolExecutor(
        max_workers=1,
        initializer=_processpool_initializer,
        initargs=(logger,),
        mp_context=multiprocessing.get_context("spawn"),
    )

    futures = {}
    for input in ["a", "b", "c"]:
        futures[executor.submit(upper, input)] = input

    for future in as_completed(futures):
        input = futures[future]
        r = future.result()
        print(f"Result: {r}", flush=True)

if __name__ == "__main__":
    main()

This is the output of the above program:

before failure
Traceback (most recent call last):
  File "/workspaces/plus/python/pdf-anonymizer/scripts/tmp/processpool-loguru.py", line 42, in <module>
    main()
  File "/workspaces/plus/python/pdf-anonymizer/scripts/tmp/processpool-loguru.py", line 37, in main
    r = future.result()
        ^^^^^^^^^^^^^^^
  File "/home/vscode/.rye/py/cpython line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.rye/py/cpython line 401, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

I am on version:

loguru==0.7.2

With a bit of debugging into loguru, I was able to determine that the subprocesses seems to be crashing on this line: )

[mayil-ai[bot]]

Some past issues to consider:

• Inconsistent cross platform behavior when using multiprocessing in spawn mode (hashtag746) | Similar issue with multiprocessing 'spawn' mode on Linux causing crashes with Loguru.

To resolve the concurrent.futures.process.BrokenProcessPool error when using Loguru with enqueue=True in a multiprocessing context on Linux, ensure that both the logger and the subprocesses use the same "spawn" context. Here are two approaches:

• Set the Start Method Globally: Set the start method to "spawn" before configuring the logger. This ensures that the logger and the subprocesses use the same context.

  import multiprocessing
  import sys
  from concurrent.futures import ProcessPoolExecutor, as_completed
  from loguru import logger
  
  def _processpool_initializer(parent_logger):
    logger._core = parent_logger._core  # type: ignore
    print("before failure", flush=True)
    logger.info("BOOM")
    print("after failure", flush=True)
  
  def upper(v):
    return str(v).upper()
  
  def main():
    multiprocessing.set_start_method("spawn")  # Set the start method to "spawn"
    logger.remove()
    logger.add(sys.stdout, enqueue=True)
  
    executor = ProcessPoolExecutor(
        max_workers=1,
        initializer=_processpool_initializer,
        initargs=(logger,),
        mp_context=multiprocessing.get_context("spawn"),
    )
  
    futures = {}
    for input in ["a", "b", "c"]:
        futures[executor.submit(upper, input)] = input
  
    for future in as_completed(futures):
        input = futures[future]
        r = future.result()
        print(f"Result: {r}", flush=True)
  
  if __name__ == "__main__":
    main()

• Use the New context Argument in Loguru (Upcoming Feature): In the next version of Loguru, a new context argument will be available for logger.add(). This allows specifying the context directly when adding the logger, ensuring compatibility with the subprocesses' context.

  import multiprocessing
  import sys
  from concurrent.futures import ProcessPoolExecutor, as_completed
  from loguru import logger
  
  def _processpool_initializer(parent_logger):
    logger._core = parent_logger._core  # type: ignore
    print("before failure", flush=True)
    logger.info("BOOM")
    print("after failure", flush=True)
  
  def upper(v):
    return str(v).upper()
  
  def main():
    context = multiprocessing.get_context("spawn")
    logger.remove()
    logger.add(sys.stdout, enqueue=True, context=context)  # Use "spawn" context
  
    executor = ProcessPoolExecutor(
        max_workers=1,
        initializer=_processpool_initializer,
        initargs=(logger,),
        mp_context=context,
    )
  
    futures = {}
    for input in ["a", "b", "c"]:
        futures[executor.submit(upper, input)] = input
  
    for future in as_completed(futures):
        input = futures[future]
        r = future.result()
        print(f"Result: {r}", flush=True)
  
  if __name__ == "__main__":
    main()

By following these steps, you should be able to resolve the BrokenProcessPool error and get Loguru working with spawned subprocesses on Linux.

💡 To rerun Mayil, comment mayil-ai rerun. Mayil will incorporate any new context added to the ticket. Include details in your rerun comment to guide Mayil!