FormatAutodetectionError crashes multiprocessing.pool

[klementng]

In my own project, I've been running into issues when using the python multiprocessing library in together with pysub2.

From my observations, when the FormatAutodetectionError exception is raised, the whole multiprocessing crashes with the following trackback

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.12/multiprocessing/pool.py", line 579, in _handle_results
    task = get()
           ^^^^^
  File "/usr/local/lib/python3.12/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: FormatAutodetectionError.__init__() missing 1 required positional argument: 'formats'

Upon further investigation, the issues seem to be caused by the transferring of the exception from the thread to the main program where the exception which causes in the formats arg being lost / not passed.

Reproducible code example

import traceback
import multiprocessing
import random
import time
import pysubs2

process_count = 0

def _run_callback(result):
    global process_count
    process_count += 1

    print(process_count)

def _error_callback(error):
    print("An error has occurred")
    traceback.print_exception(type(error), error, error.__traceback__)

def _f():

    if random.randint(1, 50) == 1:
        raise pysubs2.FormatAutodetectionError("", [])

    if random.randint(1, 10) == 1:
        raise Exception("Regular exception is properly handled by multiprocessing")

    return "ABC"

with multiprocessing.Pool(4) as pool:
    for _ in range(1000):
        pool.apply_async(
            _f,
            callback=_run_callback,
            error_callback=_error_callback,
        )

    pool.close()
    pool.join()

Related stackoverflow issue: https://stackoverflow.com/questions/70883678/python-multiprocessing-get-hung

---

My fix for this issue is to simply passing the ext into the _format kwargs to prevent the autodetection function from running

pysubs2.load(path, format_=ext)

[tkarabela]

@klementng Thanks for the report, I'll look into it.

[klementng]

The following is most likely what is going:

1. Exception is correctly raised in the thread/process with no issues

   FormatAutodetectionError("content", ['srt','ass'])

2. multiprocessing library catches the exception and transfer it to the main program
3. multiprocessing library attempt to reinitialize the exception object with only first arg of the old exception which fails (as it now missing one required positional argument).

   FormatAutodetectionError("content")

4. The whole multiprocessing pool crashes & stalls

This issue only occured in the versions 1.7.0 onwards where it seem that all the exceptions class for pysub2 was rewritten

---

A dirty hack is to simply add a default argument to the exception which will fix the issue but causes the error message to be misleading

class FormatAutodetectionError(Pysubs2Error):
  ....
    def __init__(self, content: str, formats: List[str] = []) -> None:
        self.content = content
        self.formats = formats
        if not formats:
            msg = "No suitable formats"
        else:
            msg = f"Multiple suitable formats ({formats!r})"
        super().__init__(msg)

New test code:

def _error_callback(error):
    print(error) # it now misleading 

    # The full trackback will shows the correct error & the misleading one as well
    # print("\n\n\n")
    # traceback.print_exception(type(error), error, error.__traceback__)

def _f():
    raise pysubs2.FormatAutodetectionError("", ["srt", "ass"])

with multiprocessing.Pool(4) as pool:
    for _ in range(2):
        pool.apply_async(
            _f,
            error_callback=_error_callback,
        )

    pool.close()
    pool.join()

which outputs

No suitable formats
No suitable formats

---

Ultimately, this seem to be an issue with the multiprocessing library where it expect all the exception to have same identity for the init function as the base exception class

[tkarabela]

Please upgrade to version 1.7.3, it should be fixed :)