Huge Louvain Preparation: more Huge Arrays

[knutwalker]

In addition to the existing HugeLongArray as replacement for long[]:

• int[] -> HugeIntArray
• double[] -> HugeDoubleArray
• T[] -> HugeObjectArray<T>

[jexp]

Can you comment/note somewhere on your perf-test of the Neo4j Batch-Importer Arrays.

[knutwalker]

@jexp I addressed most of your notes, I just missed the static methods from the Cursor/init methods.

Regarding benchmarks, I ran this PR on my laptop and got the following results:

 mvn -DskipTests -PBenchmark -pl benchmark package
java -jar benchmark/target/benchmark.jar -jvmArgs "-server -Xmx4g -Xms4g -XX:+UseG1GC" -psize=10000000 -psparseness=0.0 -pdistribution=uniform -gc true -rf json -rff long-arrays.json org.neo4j.graphalgo.utils.LongArrayBenchmark

long-arrays.json.zip

the tested arrays are:

primitve: plain Java long[] huge_paged: org.neo4j.graphalgo.core.utils.paged.HugeLongArray with the paging configuration (nested long[][]) huge_paged_cursor: using the HugeCursor instead of get/set huge_single: org.neo4j.graphalgo.core.utils.paged.HugeLongArray with a single page configuration (wrapper around long[]) huge_single_cursor: using the HugeCursor instead of get/set sparse: org.neo4j.graphalgo.core.utils.paged.SparseLongArray, similar to huge_paged, but pages can be null and r/w access includes appropriate checks offHeap: org.neo4j.unsafe.impl.batchimport.cache.OffHeapLongArray chunked: org.neo4j.unsafe.impl.batchimport.cache.DynamicLongArray

some possible observations:

• the JIT can inline the huge-single-page version to be the same as plain primitve access
• the same goes for cursor based access on the huge-paged version, otherwise its 30%-80% slower
• reads for the off-heap array is good, but writes are rather slow and very inconsistent
• sparse array just slightly slower than the huge-paged without cursor (as expected)

results might be something different on other machines with different CPUs, of course.