ShiftLeftSecurity / overflowdb

ShiftLeft OverflowDB
Apache License 2.0
112 stars 21 forks source link

try to fix ConcurrentModificationException #319

Closed mpollmeier closed 2 years ago

mpollmeier commented 2 years ago

We ran into a random ConcurrentModificationException:

joern-export --repr=all --format=graphml cpg.bin

Exception in thread "main" java.util.ConcurrentModificationException
  at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1661)
  at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
  at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
  at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
  at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
  at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
  at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
  at overflowdb.storage.NodesWriter.writeAndClearBatched(NodesWriter.java:41)
  at overflowdb.ReferenceManager.clearAllReferences(ReferenceManager.java:155)
  at overflowdb.Graph.shutdownNow(Graph.java:233)
  at overflowdb.Graph.close(Graph.java:219)
  at io.shiftleft.codepropertygraph.generated.Cpg.close(Cpg.scala:63)
  at scala.util.Using$Releasable$AutoCloseableIsReleasable$.release(Using.scala:391)
  at scala.util.Using$Releasable$AutoCloseableIsReleasable$.release(Using.scala:390)
  at scala.util.Using$.resource(Using.scala:267)
  at io.joern.joerncli.JoernExport$.$anonfun$new$5(JoernExport.scala:63)

I.e. the graph was modified while it was in shutdown. The reported error doesn't tell us exactly what kind of modification happened, so I went through all places and found only one that wasn't synchronized on the NodesList instance. I also added volatile to the Graph.closed flag, not sure if this helps...