ShiftLeftSecurity / overflowdb

ShiftLeft OverflowDB
Apache License 2.0
112 stars 21 forks source link

ConcurrentModificationException #424

Closed jubatusd closed 7 months ago

jubatusd commented 7 months ago

Describe the bug I encountered the ConcurrentModificationException issue in overflowDb while using joern-parse.

Stack trace:

java.util.ConcurrentModificationException
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1631)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
        at overflowdb.storage.NodesWriter.writeAndClearBatched(NodesWriter.java:41)
        at overflowdb.ReferenceManager.clearAllReferences(ReferenceManager.java:155)
        at overflowdb.Graph.shutdownNow(Graph.java:236)
        at overflowdb.Graph.close(Graph.java:221)
        at io.shiftleft.codepropertygraph.generated.Cpg.close(Cpg.scala:64)
        at io.joern.joerncli.JoernParse$.applyDefaultOverlays$$anonfun$1(JoernParse.scala:166)
        at scala.util.Try$.apply(Try.scala:210)
        at io.joern.joerncli.JoernParse$.applyDefaultOverlays(JoernParse.scala:169)
        at io.joern.joerncli.JoernParse$.run$$anonfun$2$$anonfun$1$$anonfun$1(JoernParse.scala:91)
        at scala.util.Success.flatMap(Try.scala:258)
        at io.joern.joerncli.JoernParse$.run$$anonfun$2$$anonfun$1(JoernParse.scala:92)
        at scala.util.Success.flatMap(Try.scala:258)
        at io.joern.joerncli.JoernParse$.run$$anonfun$2(JoernParse.scala:92)
        at scala.util.Success.flatMap(Try.scala:258)
        at io.joern.joerncli.JoernParse$.run(JoernParse.scala:92)
        at io.joern.joerncli.JoernParse$.run$$anonfun$1(JoernParse.scala:78)
        at scala.util.Success.flatMap(Try.scala:258)
        at io.joern.joerncli.JoernParse$.run(JoernParse.scala:78)
        at io.joern.joerncli.JoernParse$.main(JoernParse.scala:20)
        at io.joern.joerncli.JoernParse.main(JoernParse.scala)

To Reproduce To reliably reproduce this bug, there needs to be a limit on the machine's running memory. If the running memory is too small (for example, less than 32G), an OutOfMemory error will be thrown. If the running memory is large (for example, not less than 64G), then the ConcurrentModificationException (CME) will not occur. Here are the reproduction steps:

  1. Get sample code: https://github.com/linghuzhangmen/ffplay_VS2019.git
  2. Run joern-parse: ./joern-parse --max-num-def=100 /home/luo/testCodes/ffplay_VS2019/ffplay_vs2019/ffmpeg

Desktop OS: Ubuntu 20.04 Joern Version: v2.0.299 Java version: openjdk 19