JEP THREAD WARNING: Unsafe reuse of thread ForkJoinPool.commonPool-worker-19 for another Python sub-interpreter

[jiechenjiechen]

I am using the jshell as described in the README to analyze two python files. The first one cannot be parsed while the second one can. However, due to the warning stated in the title of this issue, if I parse them in the same session, the second one also fails. Command line under jshell:

import de.fraunhofer.aisec.cpg.TranslationConfiguration;
import de.fraunhofer.aisec.cpg.TranslationManager;
import de.fraunhofer.aisec.cpg.graph.declarations.FunctionDeclaration;
import de.fraunhofer.aisec.cpg.ExperimentalPython;
import de.fraunhofer.aisec.cpg.frontends.python.PythonLanguageFrontend;

var path = Paths.get("/Users/chenjie/SDG/s001603790.py");
var translationConfiguration = TranslationConfiguration.builder().sourceLocations(path.toFile()).defaultPasses().defaultLanguages().debugParser(true);
translationConfiguration.registerLanguage(PythonLanguageFrontend.class,PythonLanguageFrontend.PY_EXTENSIONS);
var config = translationConfiguration.build();
var analyzer = TranslationManager.builder().config(config).build();
var result = analyzer.analyze().get();

var path = Paths.get("/Users/chenjie/SDG/counter.py");
var translationConfiguration = TranslationConfiguration.builder().sourceLocations(path.toFile()).defaultPasses().defaultLanguages().debugParser(true);
translationConfiguration.registerLanguage(PythonLanguageFrontend.class,PythonLanguageFrontend.PY_EXTENSIONS);
var config = translationConfiguration.build();
var analyzer = TranslationManager.builder().config(config).build();
var result = analyzer.analyze().get();

Output, including error messages:

path ==> /Users/chenjie/SDG/s001603790.py
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
translationConfiguration ==> de.fraunhofer.aisec.cpg.TranslationConfiguration$Builder@6f45df59
$8 ==> de.fraunhofer.aisec.cpg.TranslationConfiguration$Builder@6f45df59
config ==> de.fraunhofer.aisec.cpg.TranslationConfiguration@76ed1b7c
analyzer ==> de.fraunhofer.aisec.cpg.TranslationManager@2b4bac49
jep.JepException: <class 'SyntaxError'>: ('invalid syntax', ('/Users/chenjie/SDG/s001603790.py', 10, 11, '    print arr[i]\n'))
    at /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.parse(ast.py:50)
    at /Users/chenjie/lab/SDG/cpg/cpg-neo4j/build/install/cpg-neo4j/./cpg.parse_code(cpg.py:1258)
    at jep.Jep.invoke(Native Method)
    at jep.Jep.invoke(Jep.java:396)
    at de.fraunhofer.aisec.cpg.frontends.python.PythonLanguageFrontend.parseInternal(PythonLanguageFrontend.kt:107)
    at de.fraunhofer.aisec.cpg.frontends.python.PythonLanguageFrontend.parse(PythonLanguageFrontend.kt:49)
    at de.fraunhofer.aisec.cpg.TranslationManager.runFrontends(TranslationManager.java:239)
    at de.fraunhofer.aisec.cpg.TranslationManager.lambda$analyze$0(TranslationManager.java:98)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1764)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1756)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1016)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1665)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1598)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
18:51:04.037 [ForkJoinPool.commonPool-worker-19] ERROR de.fraunhofer.aisec.cpg.TranslationManager - An error occurred during parsing of s001603790.py: Python failed with message: jep.JepException: <class 'SyntaxError'>: ('invalid syntax', ('/Users/chenjie/lab/SDG/s001603790.py', 10, 11, '    print arr[i]\n'))
result ==> TranslationResult[id=<null>,name=,location=<null>,argumentIndex=0]
path ==> /Users/chenjie/lab/SDG/counter.py
translationConfiguration ==> de.fraunhofer.aisec.cpg.TranslationConfiguration$Builder@6c779568
$14 ==> de.fraunhofer.aisec.cpg.TranslationConfiguration$Builder@6c779568
config ==> de.fraunhofer.aisec.cpg.TranslationConfiguration@2cdd0d4b
analyzer ==> de.fraunhofer.aisec.cpg.TranslationManager@7e9131d5
jep.JepException: JEP THREAD WARNING: Unsafe reuse of thread ForkJoinPool.commonPool-worker-19 for another Python sub-interpreter.
Please close() the previous Jep instance to ensure stability.
    at jep.Jep.<init>(Jep.java:263)
    at jep.Jep.<init>(Jep.java:251)
    at jep.SubInterpreter.<init>(SubInterpreter.java:63)
    at de.fraunhofer.aisec.cpg.frontends.python.PythonLanguageFrontend.parseInternal(PythonLanguageFrontend.kt:99)
    at de.fraunhofer.aisec.cpg.frontends.python.PythonLanguageFrontend.parse(PythonLanguageFrontend.kt:49)
    at de.fraunhofer.aisec.cpg.TranslationManager.runFrontends(TranslationManager.java:239)
    at de.fraunhofer.aisec.cpg.TranslationManager.lambda$analyze$0(TranslationManager.java:98)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1764)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1756)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1016)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1665)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1598)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
18:51:04.181 [ForkJoinPool.commonPool-worker-19] ERROR de.fraunhofer.aisec.cpg.TranslationManager - An error occurred during parsing of counter.py: Python failed with message: jep.JepException: JEP THREAD WARNING: Unsafe reuse of thread ForkJoinPool.commonPool-worker-19 for another Python sub-interpreter.
Please close() the previous Jep instance to ensure stability.
result ==> TranslationResult[id=<null>,name=,location=<null>,argumentIndex=0]

[jiechenjiechen]

Any luck? @maximiliankaul @oxisto

[jiechenjiechen]

It appears that the problem persists. @maximiliankaul

[maximiliankaul]

I cannot reproduce this with the current version of cpg-language-python and cpg-console. Forcing a crash and then parsing a second file seems to work fine.