microth
commented
7 years ago Hi Vishnu,
There is nothing strange or unexpected about this error message. As you have realized yourself, the model is quite big and a lot of heap space is required to load/run it (even if your IDE doesn't show it). I usually set the max limit to 40gb.
Cheers, Michael
Hey, @microth I've recently started using the PathLSTM, for an application which requires srl, but the sheer size of the model file ( srl-ACL2016-eng.model || 2.7G ), throws me an insufficient heap space - Out of memory error
Loading pipeline from C:\Users\Vyso\Downloads\NLP\SRL\SEMAFOR\absSemafor\LTH\wttv\PathLSTM-pre- illinois-built\srl-ACL2016-eng.model Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.lang.reflect.Array.newInstance(Array.java:75) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1883) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1529) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1919) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1529) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1919) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1529) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
Process finished with exit code 1
So naturally, I changed the heap space of both the JVM on my system, and my IDE, where I could change the vmoptions, as follows. (It was -Xms128m and -Xmx512m, by default).
custom IntelliJ IDEA VM options
-Xms2048m -Xmx4000m -XX:ReservedCodeCacheSize=240m -XX:+UseConcMarkSweepGC -XX:SoftRefLRUPolicyMSPerMB=50 -ea -Dsun.io.useCanonCaches=false -Djava.net.preferIPv4Stack=true -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow
But even after assigning around 4G as Max Heap space, I get the error. Funny thing is, in the memory management toolbar of my IDE, I can see that the code uses a max of just 500m, during runtime, so I really don't know how this heap space error is still getting thrown.
Could you tell me if this is unusual an error with this algorithm, or if you've seen it before, too? Also, should I add all the mentioned dependencies just for the parse class to function? Is there a probability that this error gets thrown because of incomplete dependency additions?
Maybe it's just a beginner level mistake from my side, but I've been trying to get out out of this problem for quite a few days now, and I'd really appreciate it if you could instruct me on how I could get rid of this error.
Thank You, Vishnu