Building a pool in the multiprocessing module crashes graalpython

[xiaxinmeng]

Building a pool in the multiprocessing module crashes graalpython with an Internal GraalVM error. See the following example "test.py".

test.py

import multiprocessing
pool = multiprocessing.Pool()
pool.map(len, [], chunksize=1)

Error message:

ERROR: java.lang.IllegalStateException: Using setAllowActions is only permitted during finalization of a language. See TruffleLanguage.finalizeContext(Object) for further details.
org.graalvm.polyglot.PolyglotException: java.lang.IllegalStateException: Using setAllowActions is only permitted during finalization of a language. See TruffleLanguage.finalizeContext(Object) for further details.
    at org.graalvm.truffle/com.oracle.truffle.api.impl.ThreadLocalHandshake$TruffleSafepointImpl.setAllowActions(ThreadLocalHandshake.java:699)
    at com.oracle.truffle.llvm.runtime.LLVMContext.finalizeContext(LLVMContext.java:592)
    at com.oracle.truffle.llvm.runtime.LLVMLanguage.finalizeContext(LLVMLanguage.java:533)
    at com.oracle.truffle.llvm.runtime.LLVMLanguage.finalizeContext(LLVMLanguage.java:93)
    at org.graalvm.truffle/com.oracle.truffle.api.LanguageAccessor$LanguageImpl.finalizeContext(LanguageAccessor.java:326)
    at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotLanguageContext.finalizeContext(PolyglotLanguageContext.java:404)
    at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotContextImpl.finalizeContext(PolyglotContextImpl.java:2925)
    at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotContextImpl.finishClose(PolyglotContextImpl.java:2473)
    at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotContextImpl.closeImpl(PolyglotContextImpl.java:2387)
    at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotContextImpl.closeAndMaybeWait(PolyglotContextImpl.java:1619)
    at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotContextImpl.close(PolyglotContextImpl.java:1554)
    at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotContextDispatch.close(PolyglotContextDispatch.java:73)
    at org.graalvm.sdk/org.graalvm.polyglot.Context.close(Context.java:831)
    at org.graalvm.sdk/org.graalvm.polyglot.Context.close(Context.java:857)
    at com.oracle.graal.python.shell.GraalPythonMain.launch(GraalPythonMain.java:681)
    at org.graalvm.launcher.AbstractLanguageLauncher.launch(AbstractLanguageLauncher.java:296)
    at org.graalvm.launcher.AbstractLanguageLauncher.launch(AbstractLanguageLauncher.java:121)
    at org.graalvm.launcher.AbstractLanguageLauncher.runLauncher(AbstractLanguageLauncher.java:168)
Caused by: Attached Guest Language Frames (0)
Internal GraalVM error, please report at https://github.com/oracle/graal/issues/.

Enviornment:

'graalpy-22.3.1-linux-amd64/bin/graalpy' Python 3.8.5 [Graal, GraalVM CE, Java 19.0.2] on Ubuntu 18.04

[msimacek]

You're not using the multiprocessing module correctly. It happens to work on CPython on Linux, but it would crash on CPython too if you run it on a platform that doesn't have fork, for example on Windows. The correct usage is this:

import multiprocessing
if __name__ == '__main__':
    pool = multiprocessing.Pool()
    pool.map(len, [], chunksize=1)

The reason why you need the if is that on implementations that don't have fork, the module gets imported again in the subprocesses and that runs the top-level code again, so it would go into infinite recursion.

The error is confusing, though, we should improve the error message.

[xiaxinmeng]

Thanks. I tried this case again with "if name == 'main':". But the crash still occurs. I do not think the root cause lies in the 'if' the statement. This crash may be related to the way we run graalpython. If we run this case in the interactive mode, no crash is detected. However, if we run this case with command " 'graalpy-22.3.1-linux-amd64/bin/graalpy' test.py" in the console, the crash occurs even with the "if" statement. I also tried this case with 'graalpy-22.3.1-macos-amd64/bin/graalpy' on macOS Big Sur 11.7.1. The behaviors are the same.

The minimal case:

import multiprocessing
if __name__ == '__main__':
    pool = multiprocessing.Pool()

System: Linux xxm 5.3.0-53-generic #47~18.04.1-Ubuntu SMP Thu May 7 13:10:50 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[msimacek]

I couldn't reproduce it, but it could depend on timing. Maybe it's related to the finalization of the pool, that the subprocesses are not properly shut down. When you run something in the pool, do the subprocesses actually finish the work and crash only when exitting? Or do they crash before/during the work? Could you also please try use the pool as context manager (with multiprocessing.Pool() as pool:)?

[xiaxinmeng]

Thanks, I tried it with "with multiprocessing.Pool() as pool:". The interpreter could work well. No crash occurs on both mac and ubuntu systems .

import multiprocessing
if __name__ == '__main__':
    with multiprocessing.Pool() as pool:
        pool.map(len, [], chunksize=1)