search

Fraunhofer-AISEC / cpg

A library to extract Code Property Graphs from C/C++, Java, Go, Python, Ruby and every other language through LLVM-IR.
https://fraunhofer-aisec.github.io/cpg/
Apache License 2.0
246 stars 59 forks source link

Python ExpressionHandler does not handle ASTSubscript #1542

Closed immqu closed 3 weeks ago

immqu commented 3 weeks ago

I'm using the Cloud Property Graph (using cpg v8.1.2) to analyze Python code. The code is contained in this directory: https://github.com/clouditor/cloud-property-graph/tree/main/ppg-testing-library/Detectability/D4-detectable-at-storage/Python

The analysis fails with the stack trace below, since the ExpressionHandler cannot handle a Python.ASTSubscript

Stack trace


kotlin.NotImplementedError: An operation is not implemented.
java.util.concurrent.ExecutionException: kotlin.NotImplementedError: An operation is not implemented.
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
    at io.clouditor.graph.App.doTranslate(App.kt:179)
    at io.clouditor.graph.App.call(App.kt:91)
    at io.clouditor.graph.TestUtilKt.executePPGAndQuery(TestUtil.kt:24)
    at io.clouditor.graph.DetectabilityTest.testD4Python(DetectabilityTest.kt:128)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
    at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
    at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:112)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40)
    at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:60)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
    at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
    at jdk.proxy2/jdk.proxy2.$Proxy5.processTestClass(Unknown Source)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
    at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
    at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
    at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
    at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
    at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: kotlin.NotImplementedError: An operation is not implemented.
    at de.fraunhofer.aisec.cpg.frontends.python.ExpressionHandler.handleNode(ExpressionHandler.kt:48)
    at de.fraunhofer.aisec.cpg.frontends.python.ExpressionHandler.handleNode(ExpressionHandler.kt:34)
    at de.fraunhofer.aisec.cpg.frontends.python.PythonHandler.handle(PythonHandler.kt:43)
    at de.fraunhofer.aisec.cpg.frontends.python.ExpressionHandler.handleDict(ExpressionHandler.kt:88)
    at de.fraunhofer.aisec.cpg.frontends.python.ExpressionHandler.handleNode(ExpressionHandler.kt:44)
    at de.fraunhofer.aisec.cpg.frontends.python.ExpressionHandler.handleNode(ExpressionHandler.kt:34)
    at de.fraunhofer.aisec.cpg.frontends.python.PythonHandler.handle(PythonHandler.kt:43)
    at de.fraunhofer.aisec.cpg.frontends.python.StatementHandler.handleAssign(StatementHandler.kt:138)
    at de.fraunhofer.aisec.cpg.frontends.python.StatementHandler.handleNode(StatementHandler.kt:48)
    at de.fraunhofer.aisec.cpg.frontends.python.StatementHandler.handleNode(StatementHandler.kt:40)
    at de.fraunhofer.aisec.cpg.frontends.python.PythonHandler.handle(PythonHandler.kt:43)
    at de.fraunhofer.aisec.cpg.frontends.python.StatementHandler.makeBlock(StatementHandler.kt:391)
    at de.fraunhofer.aisec.cpg.frontends.python.StatementHandler.makeBlock$default(StatementHandler.kt:388)
    at de.fraunhofer.aisec.cpg.frontends.python.StatementHandler.handleFunctionDef(StatementHandler.kt:333)
    at de.fraunhofer.aisec.cpg.frontends.python.StatementHandler.handleFunctionDef$default(StatementHandler.kt:193)
    at de.fraunhofer.aisec.cpg.frontends.python.StatementHandler.handleNode(StatementHandler.kt:45)
    at de.fraunhofer.aisec.cpg.frontends.python.StatementHandler.handleNode(StatementHandler.kt:40)
    at de.fraunhofer.aisec.cpg.frontends.python.PythonHandler.handle(PythonHandler.kt:43)
    at de.fraunhofer.aisec.cpg.frontends.python.PythonLanguageFrontend.pythonASTtoCPG(PythonLanguageFrontend.kt:263)
    at de.fraunhofer.aisec.cpg.frontends.python.PythonLanguageFrontend.parse(PythonLanguageFrontend.kt:79)
    at de.fraunhofer.aisec.cpg.TranslationManager.parse(TranslationManager.kt:353)
    at de.fraunhofer.aisec.cpg.TranslationManager.parseSequentially(TranslationManager.kt:306)
    at de.fraunhofer.aisec.cpg.TranslationManager.runFrontends(TranslationManager.kt:225)
    at de.fraunhofer.aisec.cpg.TranslationManager.analyzeNonAsync(TranslationManager.kt:87)
    at de.fraunhofer.aisec.cpg.TranslationManager.analyze$lambda$0(TranslationManager.kt:69)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1760)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)```
oxisto commented 3 weeks ago

Its probably coming from this line: https://github.com/clouditor/cloud-property-graph/blob/6c7bc074386531bd18ff83e9a2656c1f266839cc/ppg-testing-library/Detectability/D4-detectable-at-storage/Python/server.py#L16

maximiliankaul commented 3 weeks ago

@immqu please see #1545 Does this fix you issue? There are still open points like ast.slice missing...

oxisto commented 3 weeks ago

Closed by #1545