search

Fraunhofer-AISEC / cpg

A library to extract Code Property Graphs from C/C++, Java, Go, Python, Ruby and every other language through LLVM-IR.
https://fraunhofer-aisec.github.io/cpg/
Apache License 2.0
263 stars 60 forks source link

Concurrent Modification Exception in ScopeManager #681

Closed anon767 closed 2 years ago

anon767 commented 2 years ago

Hey, Im encountering a Concurrent Modification Exception caused by the ScopeManager if I get this right:

Caused by: java.util.ConcurrentModificationException
        at java.util.IdentityHashMap$IdentityHashMapIterator.nextIndex(IdentityHashMap.java:735) ~[?:?]
        at java.util.IdentityHashMap$EntryIterator.next(IdentityHashMap.java:843) ~[?:?]
        at java.util.IdentityHashMap$EntryIterator.next(IdentityHashMap.java:837) ~[?:?]
        at java.util.IdentityHashMap.putAll(IdentityHashMap.java:506) ~[?:?]
        at de.fraunhofer.aisec.cpg.passes.scopes.ScopeManager.mergeFrom(ScopeManager.kt:183) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.TranslationManager.parseParallel(TranslationManager.kt:265) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.TranslationManager.runFrontends(TranslationManager.kt:202) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.TranslationManager.analyze$lambda-2(TranslationManager.kt:86) ~[cpg-core-master-SNAPSHOT.jar:?]
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) ~[?:?]
        at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692) ~[?:?]
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) ~[?:?]
        at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) ~[?:?]
        at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) ~[?:?]
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) ~[?:?]
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) ~[?:?]

And here another related stack trace:

Caused by: java.util.ConcurrentModificationException
        at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1042) ~[?:?]
        at java.util.ArrayList$Itr.next(ArrayList.java:996) ~[?:?]
        at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1047) ~[?:?]
        at de.fraunhofer.aisec.cpg.TranslationManager.runFrontends(TranslationManager.kt:415) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.TranslationManager.analyze$lambda-2(TranslationManager.kt:86) ~[cpg-core-master-SNAPSHOT.jar:?]
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) ~[?:?]
        at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692) ~[?:?]
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) ~[?:?]
        at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) ~[?:?]
        at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) ~[?:?]
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) ~[?:?]
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) ~[?:?]

Seems related to https://github.com/Fraunhofer-AISEC/cpg/issues/651 however this time, typeSystemActiveInFrontend is explicitly set to false .

Regards, Tom

anon767 commented 2 years ago

An educated guess to circumvent the issue would be to wrap the putAll call in the scopemanager inbetween a synchronized block: https://github.com/Fraunhofer-AISEC/cpg/blob/fdd3f0c8343acf9edf03a8bdff99142d4090cfba/cpg-core/src/main/java/de/fraunhofer/aisec/cpg/passes/scopes/ScopeManager.kt#L183

or a cleaner alternative: to change the map impl to a thread safe alternative

oxisto commented 2 years ago

An educated guess to circumvent the issue would be to wrap the putAll call in the scopemanager inbetween a synchronized block:

https://github.com/Fraunhofer-AISEC/cpg/blob/fdd3f0c8343acf9edf03a8bdff99142d4090cfba/cpg-core/src/main/java/de/fraunhofer/aisec/cpg/passes/scopes/ScopeManager.kt#L183

or a cleaner alternative: to change the map impl to a thread safe alternative

What "bothers" me here, is that mergeFrom is completely sequential, there is no parallel call in that function, so I have no idea why this CME is thrown. It would be good to have something that makes this reproducable.

anon767 commented 2 years ago

The exceptions seem to disappear with .loadIncludes(false)

Edit: Nope still persists

anon767 commented 2 years ago

If I turn off parallelFrontends I get following errors in the same frequency as the ConcurrentExceptions. Is there some connection between those?

Caused by: java.lang.ClassCastException: class org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTUnaryExpression cannot be cast to class org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTFieldReference (org.eclipse.cdt.internal.core.dom.parser.cpp.CPP$
STUnaryExpression and org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTFieldReference are in unnamed module of loader 'app')
        at de.fraunhofer.aisec.cpg.frontends.cpp.ExpressionHandler.handleFunctionCallExpression(ExpressionHandler.kt:488) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.ExpressionHandler._init_$lambda-7(ExpressionHandler.kt:91) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.Handler.handle(Handler.java:114) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.ExpressionHandler.handleBinaryExpression(ExpressionHandler.kt:667) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.ExpressionHandler._init_$lambda-2(ExpressionHandler.kt:71) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.Handler.handle(Handler.java:114) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler.handleExpressionStatement(StatementHandler.kt:255) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler._init_$lambda-4(StatementHandler.kt:56) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.Handler.handle(Handler.java:114) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler.handleIfStatement(StatementHandler.kt:143) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler._init_$lambda-5(StatementHandler.kt:58) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.Handler.handle(Handler.java:114) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler.handleCompoundStatement(StatementHandler.kt:289) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler._init_$lambda-1(StatementHandler.kt:50) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.Handler.handle(Handler.java:114) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler.handleWhileStatement(StatementHandler.kt:191) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler._init_$lambda-6(StatementHandler.kt:60) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.Handler.handle(Handler.java:114) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler.handleCompoundStatement(StatementHandler.kt:289) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler._init_$lambda-1(StatementHandler.kt:50) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.Handler.handle(Handler.java:114) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.DeclarationHandler.handleFunctionDefinition(DeclarationHandler.kt:162) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.DeclarationHandler._init_$lambda-3(DeclarationHandler.kt:60) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.Handler.handle(Handler.java:114) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.DeclarationHandler.handleTranslationUnit(DeclarationHandler.kt:529) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.CXXLanguageFrontend.parse(CXXLanguageFrontend.kt:223) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.TranslationManager.parse(TranslationManager.kt:340) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.TranslationManager.parseSequentially(TranslationManager.kt:284) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.TranslationManager.runFrontends(TranslationManager.kt:204) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.TranslationManager.analyze$lambda-2(TranslationManager.kt:86) ~[cpg-core-master-SNAPSHOT.jar:?]
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) ~[?:?]
        at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692) ~[?:?]
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) ~[?:?]
        at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) ~[?:?]
        at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) ~[?:?]
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) ~[?:?]
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) ~[?:?]
oxisto commented 2 years ago

If I turn off parallelFrontends I get following errors in the same frequency as the ConcurrentExceptions. Is there some connection between those?

Caused by: java.lang.ClassCastException: class org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTUnaryExpression cannot be cast to class org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTFieldReference (org.eclipse.cdt.internal.core.dom.parser.cpp.CPP$
STUnaryExpression and org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTFieldReference are in unnamed module of loader 'app')
        at de.fraunhofer.aisec.cpg.frontends.cpp.ExpressionHandler.handleFunctionCallExpression(ExpressionHandler.kt:488) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.ExpressionHandler._init_$lambda-7(ExpressionHandler.kt:91) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.Handler.handle(Handler.java:114) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.ExpressionHandler.handleBinaryExpression(ExpressionHandler.kt:667) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.ExpressionHandler._init_$lambda-2(ExpressionHandler.kt:71) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.Handler.handle(Handler.java:114) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler.handleExpressionStatement(StatementHandler.kt:255) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler._init_$lambda-4(StatementHandler.kt:56) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.Handler.handle(Handler.java:114) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler.handleIfStatement(StatementHandler.kt:143) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler._init_$lambda-5(StatementHandler.kt:58) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.Handler.handle(Handler.java:114) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler.handleCompoundStatement(StatementHandler.kt:289) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler._init_$lambda-1(StatementHandler.kt:50) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.Handler.handle(Handler.java:114) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler.handleWhileStatement(StatementHandler.kt:191) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler._init_$lambda-6(StatementHandler.kt:60) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.Handler.handle(Handler.java:114) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler.handleCompoundStatement(StatementHandler.kt:289) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.StatementHandler._init_$lambda-1(StatementHandler.kt:50) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.Handler.handle(Handler.java:114) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.DeclarationHandler.handleFunctionDefinition(DeclarationHandler.kt:162) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.DeclarationHandler._init_$lambda-3(DeclarationHandler.kt:60) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.Handler.handle(Handler.java:114) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.DeclarationHandler.handleTranslationUnit(DeclarationHandler.kt:529) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.frontends.cpp.CXXLanguageFrontend.parse(CXXLanguageFrontend.kt:223) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.TranslationManager.parse(TranslationManager.kt:340) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.TranslationManager.parseSequentially(TranslationManager.kt:284) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.TranslationManager.runFrontends(TranslationManager.kt:204) ~[cpg-core-master-SNAPSHOT.jar:?]
        at de.fraunhofer.aisec.cpg.TranslationManager.analyze$lambda-2(TranslationManager.kt:86) ~[cpg-core-master-SNAPSHOT.jar:?]
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) ~[?:?]
        at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692) ~[?:?]
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) ~[?:?]
        at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) ~[?:?]
        at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) ~[?:?]
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) ~[?:?]
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) ~[?:?]

Doubtful, but this is an easy fix: https://github.com/Fraunhofer-AISEC/cpg/pull/687. Can you try it with that branch? Maybe the other ones will disappear magically :D

anon767 commented 2 years ago

Doubtful, but this is an easy fix: #687. Can you try it with that branch? Maybe the other ones will disappear magically :D

Yay that fixed it :)

oxisto commented 2 years ago

Doubtful, but this is an easy fix: #687. Can you try it with that branch? Maybe the other ones will disappear magically :D Yay that fixed it :)

Does it fix the concurrent modification exceptions as well?

anon767 commented 2 years ago

Hm I ran my program several times and the Exception is not appearing yet. So it may be fixed or I just havn't encountered it yet