`exit` inside a loky-submitted function causes the main process to exit

[sjdv1982]

Hello, I am looking into loky as a replacement for multiprocessing. However, my functions often call C extensions, and I couldn't find an easy equivalent for mp.Process.exitcode.

When trying out loky to see if I can retrieve the exit code somehow, I ran the following code snippet:

import time
import loky

executor = loky.get_reusable_executor()

def func():
    import sys
    sys.exit(1)

fut = executor.submit(func)
r = fut.result()
print("Main process still alive")
print(r)
time.sleep(3)

This exits within a fraction of a second, without printing anything.

Package versions are loky-3.4.0 cloudpickle-2.2.1 psutil-5.9.5. It happens under Python 3.9, 3.10 and 3.11. Using exit or sys.exit makes no difference.

[tomMoral]

Actually, looking back at it, the error is re-raised. This code works as expected:

import time
import loky

executor = loky.get_reusable_executor()

def func():
    import sys
    time.sleep(.5)
    print("run exit")
    sys.exit(0)

fut = executor.submit(func)
try:
    r = fut.result()
    print(r)
except SystemExit:
    print("sys exit called")
print("Main process still alive")
time.sleep(3)

Not sure what is the expected behavior here. We want to re-raise the errors from remote jobs (I think this is the same behavior as concurrent.futures) but it is true that this can be surprising.

[sjdv1982]

Ah, thank you very much. I can confirm on both counts, that concurrent.futures.ThreadPoolExecutor does the same thing, and that I am very surprised.