Loguru leaving several shared memory file descriptors dangling when using enqueue

[TiagoG234]

Issue Overview: Within a multithreaded environment utilizing Loguru for logging, there arises an issue with dangling open file descriptors, specifically of shared memory files. These descriptors accumulate rapidly, posing a risk of resource exhaustion as they approach the operating system's limit. This issue is exacerbated when multiplied by the number of active threads.

Implementation Details: The implementation utilizes a function

def create_logger(name: str, logger: Optional[Logger] = None, *, with_installation_id: bool = True) -> Logger:
    if logger is None:
        new_logger = Logger(name, with_installation_id=with_installation_id)
        return new_logger
    return logger

to instantiate the Logger class, with the option to create a new instance or reuse an existing one based on input parameters. Additionally, the method

def add_handlers(self, log_level):

        logger.add(f"{Logger._logger_folder}{self._file_name}.log", backtrace=True,
                   diagnose=True, enqueue=True, colorize=True, format=Logger.__FMT,
                   rotation="1 hour", retention="14 days", compression="gz", level=log_level,
                   filter=lambda record: record["extra"].get("file_name") == self._file_name)
        logger.add(sys.stderr, backtrace=True,
                   diagnose=True, enqueue=True, colorize=True, format=Logger.__FMT, level=log_level,
                   filter=lambda record: record["extra"].get("file_name") == self._file_name)

is employed to attach handlers to the logger, with the enqueue=True parameter set to ensure thread safety and non-blocking logging operations.

Identified Issue: The usage of enqueue=True in the add_handlers() method results in the accumulation of dangling open file descriptors, particularly for shared memory files. While intended to enhance thread safety by preventing blocking, this parameter inadvertently contributes to excessive creation of file descriptors, potentially reaching the operating system's limit.

Additional Information:

Python Version: 3.6.9
Loguru Version: 0.6.0

Am I missing something in the proper configuration or usage that would prevent this? Is this a know issue?

[Delgan]

Can you give more details about how create_logger() and add_handlers() are used exactly, please?

More specifically: are you sure that logger.add() isn't called too many times? Each time logger.add("file.log") is called, a new file handler will be created and it won't be cleaned until logger.remove() is called.

[TiagoG234]

Thank you for the prompt response.

The create_logger() acts as a Singleton of sorts, since I want to have multiple classes writing to the same log file that corresponds to a top level service. Hence, when we want to attribute a logger to a given class we use the create_logger() function to return the Logger object that was passed as an argument in the instantiation of the class, and so we ensure that we are only instantiating a single Logger object per top level service that is then passed down to each class.

I do not think the problem here is logger.add() being called too many times. I did a simple test and created a Logger object, then in a thread of the same process I logged something with the same Logger object. What I noticed was that for each new thread where I use the Logger object the number of these dangling file handlers increased without me ever instantiating a new Logger object.

This behavior was only verified when using enqueue=True.

[pauloribeiro92]

I'm facing the exact same issue with Python 3.11. Any developments here?

[Delgan]

Any chance to get a minimal reproducible example, please?

I don't see what would cause such behavior, apart from inadvertent misuse of the logger.