Closed ChetanBhasin closed 8 years ago
More information on above: the error happens with the provided model for both CoreNLPParser and FastNLPParser.
@marcovzla Please remind me why we commented out the javaOptions in build.sbt? I know we had a good reason, but I don't recall what it was.
We had issues with travis, so we stopped forking the jvm. See https://github.com/travis-ci/travis-ci/issues/3775
We are now using .sbtopts
to set the memory, but that file is ignored in windows.
Could that be the issue?
@MihaiSurdeanu Does this mean that such behavior is expected under default parameters and that one should provide more memory?
This is strange because the examples odin-examples
repository work just fine under default parameters even while using the same version of the library.
@ChetanBhasin, odin-examples
is actually using an outdated version of processors
(5.7.0):
That was before we started using Travis...
@myedibleenso I just tried updating the version in odin-examples
to 5.8.1 without making any change to the code it still looks fine.
Jumping inside REPL and trying out the example code, however, fails again.
Here, I'd paste the original input and output from the console launched using sbt console
:
scala> import edu.arizona.sista.processors.corenlp._
import edu.arizona.sista.processors.corenlp._
scala> val proc = new CoreNLPProcessor(withDiscourse = true)
proc: edu.arizona.sista.processors.corenlp.CoreNLPProcessor = edu.arizona.sista.processors.corenlp.CoreNLPProcessor@53814411
scala> val doc = proc.annotate("How many times have you been to China before?")
Adding annotator tokenize
Adding annotator ssplit
Adding annotator pos
Reading POS tagger model from edu/stanford/nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger ... done [1.1 sec].
Adding annotator lemma
Adding annotator ner
Loading classifier from edu/stanford/nlp/models/ner/english.all.3class.distsim.crf.ser.gz ... done [5.1 sec].
Loading classifier from edu/stanford/nlp/models/ner/english.muc.7class.distsim.crf.ser.gz ... done [2.6 sec].
Loading classifier from edu/stanford/nlp/models/ner/english.conll.4class.distsim.crf.ser.gz ... done [3.6 sec].
sutime.binder.1.
Initializing JollyDayHoliday for sutime with classpath:edu/stanford/nlp/models/sutime/jollyday/Holidays_sutime.xml
Reading TokensRegex rules from edu/stanford/nlp/models/sutime/defs.sutime.txt
Reading TokensRegex rules from edu/stanford/nlp/models/sutime/english.sutime.txt
Mar 22, 2016 2:06:51 PM edu.stanford.nlp.ling.tokensregex.CoreMapExpressionExtractor appendRules
INFO: Ignoring inactive rule: null
Mar 22, 2016 2:06:51 PM edu.stanford.nlp.ling.tokensregex.CoreMapExpressionExtractor appendRules
INFO: Ignoring inactive rule: temporal-composite-8:ranges
Reading TokensRegex rules from edu/stanford/nlp/models/sutime/english.holidays.sutime.txt
Loading parser from serialized file edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz ... done [0.8 sec].
macro=true
featureCountThresh=10
featureFactory=edu.arizona.sista.processors.corenlp.chunker.ChunkingFeatureFactory
Adding annotator dcoref
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.AbstractList.iterator(AbstractList.java:288)
at java.util.AbstractList.hashCode(AbstractList.java:540)
at java.util.HashMap.hash(HashMap.java:338)
at java.util.HashMap.readObject(HashMap.java:1397)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at edu.stanford.nlp.io.IOUtils.readObjectFromURLOrClasspathOrFileSystem(IOUtils.java:313)
at edu.stanford.nlp.dcoref.Dictionaries.loadGenderNumber(Dictionaries.java:389)
at edu.stanford.nlp.dcoref.Dictionaries.<init>(Dictionaries.java:553)
at edu.stanford.nlp.dcoref.Dictionaries.<init>(Dictionaries.java:462)
at edu.stanford.nlp.dcoref.SieveCoreferenceSystem.<init>(SieveCoreferenceSystem.java:283)
at edu.stanford.nlp.pipeline.DeterministicCorefAnnotator.<init>(DeterministicCorefAnnotator.java:52)
at edu.stanford.nlp.pipeline.AnnotatorImplementations.coref(AnnotatorImplementations.java:181)
at edu.stanford.nlp.pipeline.AnnotatorFactories$12.create(AnnotatorFactories.java:485)
at edu.stanford.nlp.pipeline.AnnotatorPool.get(AnnotatorPool.java:85)
at edu.stanford.nlp.pipeline.StanfordCoreNLP.construct(StanfordCoreNLP.java:289)
at edu.stanford.nlp.pipeline.StanfordCoreNLP.<init>(StanfordCoreNLP.java:126)
at edu.arizona.sista.processors.corenlp.CoreNLPProcessor.mkCoref(CoreNLPProcessor.scala:56)
at edu.arizona.sista.processors.corenlp.CoreNLPProcessor.coref$lzycompute(CoreNLPProcessor.scala:31)
at edu.arizona.sista.processors.corenlp.CoreNLPProcessor.coref(CoreNLPProcessor.scala:31)
at edu.arizona.sista.processors.corenlp.CoreNLPProcessor.resolveCoreference(CoreNLPProcessor.scala:148)
at edu.arizona.sista.processors.Processor$class.annotate(Processor.scala:108)
at edu.arizona.sista.processors.shallownlp.ShallowNLPProcessor.annotate(ShallowNLPProcessor.scala:25)
at edu.arizona.sista.processors.Processor$class.annotate(Processor.scala:88)
at edu.arizona.sista.processors.shallownlp.ShallowNLPProcessor.annotate(ShallowNLPProcessor.scala:25)
... 1 elided
Please note that the above code works very well if memory is increased (i.e, SBT launched with sbt -J-Xmx4G -J-Xms4G
).
Is this happening because of @marcovzla said: are you on Windows? We now configure sbt through .sbtopts, which seems to be ignored on Windows.
@MihaiSurdeanu No, running this on OS X. I'll try it out on CentOS now.
@MihaiSurdeanu
Update: Running across same error using CentOS.
Hmm. @myedibleenso, @marcovzla: any idea what causes this? Can you replicate it?
If this cannot be reproduced, we can just close the issue and I can update you if I find any reason for this in future.
Meanwhile, it's possible on my end to just override memory limit using flags and have the applications run.
Hi, @ChetanBhasin. Thanks for pointing this out, and I'm sorry for such a late response.
The odin-examples
project appears to be missing an .sbtopts
, so the error you reported may have been due to changes in processors
(ex. changes to a bionlp dependency such as bioresources
) that ended up requiring more memory than previous versions.
I'm going to close this issue and open one in odin-examples
for adding .sbtopts
.
@myedibleenso I see, thanks. We did get it to work by explicitly passing the options. Thanks for looking into this.
The example code from the README file for Scala keeps throwing OutOfMemoryError with default SBT parameters.
I remember that I have previously used this library without making any changes to JVM parameters and it works just fine. This is the first time, and with the current version, that I have noticed something like this.
Any idea why this might be happening?