OutOfMemoryError - Githubissues

rpx99 commented 5 days ago

Increased the size a bit to 500k / 200k...

storageManager.setRoot(gigaMap = RandomGenerator.createMap(500_000));

Creating random data ... [19.706s][warning][os,thread] Attempt to unprotect stack guard pages failed (0x000005a0c52cc000-0x000005a0c52d0000).
Exception in thread "main" java.lang.OutOfMemoryError: Unable to allocate 2064 bytes
    at java.base/jdk.internal.misc.Unsafe.allocateMemory(Unsafe.java:632)
    at jdk.unsupported/sun.misc.Unsafe.allocateMemory(Unsafe.java:462)
    at org.eclipse.serializer.memory.sun.JdkInternals.allocateMemory(JdkInternals.java:416)
    at org.eclipse.serializer.memory.sun.JdkMemoryAccessor.allocateMemory(JdkMemoryAccessor.java:89)
    at org.eclipse.serializer.memory.XMemory.allocate(XMemory.java:937)
    at one.microstream.gigamap.BitmapLevel2.allocateLevel2Block(BitmapLevel2.java:190)
    at one.microstream.gigamap.BitmapLevel2.allocateNew(BitmapLevel2.java:180)
    at one.microstream.gigamap.BitmapLevel2.<init>(BitmapLevel2.java:515)
    at one.microstream.gigamap.BitmapLevel2.New(BitmapLevel2.java:495)
    at one.microstream.gigamap.BitmapEntry.lambda$static$0(BitmapEntry.java:18)
    at one.microstream.gigamap.BitmapLevel3.ensureLevel2Segment(BitmapLevel3.java:428)
    at one.microstream.gigamap.BitmapLevel3.setCurrentAddLevel1Segment(BitmapLevel3.java:401)
    at one.microstream.gigamap.BitmapLevel3.add(BitmapLevel3.java:258)
    at one.microstream.gigamap.BitmapEntry.add(BitmapEntry.java:113)
    at one.microstream.gigamap.BitmapIndex$AbstractEntriesBased.internalAdd(BitmapIndex.java:479)
    at one.microstream.gigamap.BitmapIndex$Composite.internalAdd(BitmapIndex.java:1252)
    at one.microstream.gigamap.BitmapIndex$Composite.addAll(BitmapIndex.java:1270)
    at one.microstream.gigamap.BitmapIndices$Default.addAll(BitmapIndices.java:349)
    at one.microstream.gigamap.GigaIndices$Default.addAll(GigaIndices.java:108)
    at one.microstream.gigamap.GigaMap$Default.addAll(GigaMap.java:1466)
    at one.microstream.gigamap.examples.RandomGenerator.createMap(RandomGenerator.java:29)
    at one.microstream.gigamap.examples.BasicExample.ensureGigaMap(BasicExample.java:33)
    at one.microstream.gigamap.examples.BasicExample.<clinit>(BasicExample.java:18)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1608 bytes. Error detail: AllocateHeap
# An error report file with more information is saved as:
# /home/rpx/Downloads/gigamap/enterprise-edition-beta/examples/hs_err_pid63182.log

storageManager.setRoot(gigaMap = RandomGenerator.createMap(200_000));


Creating random data ... [9.416s][warning][os,thread] Attempt to unprotect stack guard pages failed (0x00000c9b0eb1c000-0x00000c9b0eb20000).
Exception in thread "main" java.lang.OutOfMemoryError: Unable to allocate 2064 bytes
    at java.base/jdk.internal.misc.Unsafe.allocateMemory(Unsafe.java:632)
    at jdk.unsupported/sun.misc.Unsafe.allocateMemory(Unsafe.java:462)
    at org.eclipse.serializer.memory.sun.JdkInternals.allocateMemory(JdkInternals.java:416)
    at org.eclipse.serializer.memory.sun.JdkMemoryAccessor.allocateMemory(JdkMemoryAccessor.java:89)
    at org.eclipse.serializer.memory.XMemory.allocate(XMemory.java:937)
    at one.microstream.gigamap.BitmapLevel2.allocateLevel2Block(BitmapLevel2.java:190)
    at one.microstream.gigamap.BitmapLevel2.allocateNew(BitmapLevel2.java:180)
    at one.microstream.gigamap.BitmapLevel2.<init>(BitmapLevel2.java:515)
    at one.microstream.gigamap.BitmapLevel2.New(BitmapLevel2.java:495)
    at one.microstream.gigamap.BitmapEntry.lambda$static$0(BitmapEntry.java:18)
    at one.microstream.gigamap.BitmapLevel3.ensureLevel2Segment(BitmapLevel3.java:428)
    at one.microstream.gigamap.BitmapLevel3.setCurrentAddLevel1Segment(BitmapLevel3.java:401)
    at one.microstream.gigamap.BitmapLevel3.add(BitmapLevel3.java:258)
    at one.microstream.gigamap.BitmapEntry.add(BitmapEntry.java:113)
    at one.microstream.gigamap.BitmapIndex$Default.addAll(BitmapIndex.java:617)
    at one.microstream.gigamap.BitmapIndices$Default.addAll(BitmapIndices.java:349)
    at one.microstream.gigamap.GigaIndices$Default.addAll(GigaIndices.java:108)
    at one.microstream.gigamap.GigaMap$Default.addAll(GigaMap.java:1466)
    at one.microstream.gigamap.examples.RandomGenerator.createMap(RandomGenerator.java:29)
    at one.microstream.gigamap.examples.BasicExample.ensureGigaMap(BasicExample.java:33)
    at one.microstream.gigamap.examples.BasicExample.<clinit>(BasicExample.java:18)

Wonder why it cannot handle that quite small sizes. I have 32 GB in my system. Is it an Intellij limit?

@fh-ms Do you have the same problem? Expected it to be capable to manage billions of records.

Anything I do wrongly?

hs_err_pid63182.log

fh-ms commented 5 days ago

We have tested the GigaMap with billions of elements.

Could you please provide a complete reproducible code example? Or was it the example code?

What JVM/JDK did you use?

rpx99 commented 5 days ago

I used the example code you provided. It works with the size of 10.000 and 100.000, too. But for larger instances not.

JVM / jdk is in the log file I attached.

microstream-one / enterprise-edition-beta

OutOfMemoryError #10