search

ldbc / ldbc_graphalytics

Generic driver for LDBC Graphalytics implementation
https://ldbcouncil.org/benchmarks/graphalytics/
Apache License 2.0
81 stars 34 forks source link

Prevent overflows with bigger HashSets #192

Closed hegyibalint closed 5 years ago

hegyibalint commented 5 years ago

With XL size graphs, the OpenHashSets can reach their 2^31 limitation.

Failed to validate benchmark run.
java.lang.IllegalArgumentException: Too large (1110540106 expected elements with load factor 0.75)
        at it.unimi.dsi.fastutil.HashCommon.arraySize(HashCommon.java:227) ~[graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]
        at it.unimi.dsi.fastutil.longs.LongOpenHashSet.<init>(LongOpenHashSet.java:85) ~[graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]
        at it.unimi.dsi.fastutil.longs.LongOpenHashSet.<init>(LongOpenHashSet.java:95) ~[graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]
        at science.atlarge.graphalytics.validation.LongVertexValidator.validate(LongVertexValidator.java:81) ~[graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]
        at science.atlarge.graphalytics.execution.BenchmarkRunner.validate(BenchmarkRunner.java:168) ~[graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]
        at science.atlarge.graphalytics.execution.RunnerService.onReceive(RunnerService.java:173) [graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]
        at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) [graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]
        at akka.actor.Actor$class.aroundReceive(Actor.scala:539) [graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]
        at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) [graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:612) [graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]
        at akka.actor.ActorCell.invoke(ActorCell.scala:581) [graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) [graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]
        at akka.dispatch.Mailbox.run(Mailbox.scala:229) [graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]
        at akka.dispatch.Mailbox.exec(Mailbox.scala:241) [graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [graphalytics-platforms-graphblas-0.1-SNAPSHOT-default.jar:?]

The OpenHashBigSets has capacity to handle huge amount of keys, therefore it's a sensible choice to replace the OpenHashSets.

szarnyasg commented 5 years ago

This was a serious limitation - many thanks for the fix!