AngledLuffa
commented
4 years ago MUC is smaller than all or conll in terms of file size, but it has 7 classes, so this may increase the memory footprint when loaded. What happens if you only load the MUC model?
Perhaps there is something silently failing when loading that model in a low memory environment.
On Sun, Oct 27, 2019 at 3:49 PM erksch notifications@github.com wrote:
I (somewhat) successfully integrated CoreNLP in an Android app. The following annotator configuration works just fine:
props.setProperty("annotators", "tokenize,ssplit,pos,lemma")
But as soon as I add the NER annotator I start to get the following error:
Caused by: java.lang.NullPointerException: Attempt to invoke interface method 'int java.util.List.size()' on a null object reference at edu.stanford.nlp.util.HashIndex.size(HashIndex.java:94) at edu.stanford.nlp.ie.crf.CRFClassifier.getCliqueTree(CRFClassifier.java:1499) at edu.stanford.nlp.ie.crf.CRFClassifier.getSequenceModel(CRFClassifier.java:1190) at edu.stanford.nlp.ie.crf.CRFClassifier.getSequenceModel(CRFClassifier.java:1186) at edu.stanford.nlp.ie.crf.CRFClassifier.classifyMaxEnt(CRFClassifier.java:1218) at edu.stanford.nlp.ie.crf.CRFClassifier.classify(CRFClassifier.java:1128) at edu.stanford.nlp.ie.AbstractSequenceClassifier.classifySentence(AbstractSequenceClassifier.java:299) at edu.stanford.nlp.ie.ClassifierCombiner.classify(ClassifierCombiner.java:476) at edu.stanford.nlp.ie.NERClassifierCombiner.classifyWithGlobalInformation(NERClassifierCombiner.java:269) at edu.stanford.nlp.ie.AbstractSequenceClassifier.classifySentenceWithGlobalInformation(AbstractSequenceClassifier.java:343) at edu.stanford.nlp.pipeline.NERCombinerAnnotator.doOneSentence(NERCombinerAnnotator.java:368) at edu.stanford.nlp.pipeline.SentenceAnnotator.annotate(SentenceAnnotator.java:102) at edu.stanford.nlp.pipeline.NERCombinerAnnotator.annotate(NERCombinerAnnotator.java:310) at edu.stanford.nlp.pipeline.AnnotationPipeline.annotate(AnnotationPipeline.java:76) at edu.stanford.nlp.pipeline.StanfordCoreNLP.annotate(StanfordCoreNLP.java:637) at edu.stanford.nlp.pipeline.StanfordCoreNLP.annotate(StanfordCoreNLP.java:629)
The code I use (Kotlin):
val props = Properties() props.setProperty("annotators", "tokenize,ssplit,pos,lemma,ner") pipeline = StanfordCoreNLP(props)val document = CoreDocument("Joe Smith is from Seattle.") pipeline.annotate(document)
The error is very similar to the one described in this issue https://github.com/stanfordnlp/CoreNLP/issues/861 where the author tried to use the parser annotator. Debugging
I debugged the stack trace and found that the error is caused by this line (on classIndex.size()) in CRFClassifier:1480 https://github.com/stanfordnlp/CoreNLP/blob/eb43d5d9150de97f8061fa06b838f1d021586789/src/edu/stanford/nlp/ie/crf/CRFClassifier.java#L1480 :
return CRFCliqueTree.getCalibratedCliqueTree(data, labelIndices, classIndex.size(), classIndex, flags.backgroundSymbol, getCliquePotentialFunctionForTest(), featureVal);
Meaning classIndex is null and must have not been initialized properly.
The classIndex property of CRFClassifier is initialized in the loadClassifier(ObjectInputStream ois, Properties props) https://github.com/stanfordnlp/CoreNLP/blob/eb43d5d9150de97f8061fa06b838f1d021586789/src/edu/stanford/nlp/ie/crf/CRFClassifier.java#L2570 method:
public void loadClassifier(ObjectInputStream ois, Properties props) { Object o = ois.readObject(); [...] classIndex = (Index
) ois.readObject(); I found out that the passed ObjectInputStream is effectively a stream on the file from a model path that is determined in the NERCombinerAnnotator constructor:
public NERCombinerAnnotator(Properties properties) throws IOException { List
models = new ArrayList<>(); String modelNames = properties.getProperty("ner.model"); if (modelNames == null) { modelNames = DefaultPaths.DEFAULT_NER_THREECLASS_MODEL + ',' + DefaultPaths.DEFAULT_NER_MUC_MODEL + ',' + DefaultPaths.DEFAULT_NER_CONLL_MODEL; } [...] String[] loadPaths = models.toArray(new String[models.size()]); Those loadPaths are iterated in the loadClassifiers method in ClassifierCombiner:
private void loadClassifiers(Properties props, List
paths) throws IOException { baseClassifiers = new ArrayList<>(); [...] for(String path: paths) { AbstractSequenceClassifier cls = loadClassifierFromPath(props, path); baseClassifiers.add(cls); [...] } When adding a breakpoint to this method I found out that the first model path (DefaultPaths.DEFAULT_NER_THREECLASS_MODEL) in the first iteration of the for-loop is loaded without problems and the classIndex property is set correctly:
[image: Bildschirmfoto vom 2019-10-27 23-39-33] https://user-images.githubusercontent.com/19290349/67643066-2a3f5800-f913-11e9-93a5-d4a9c44b8b02.png
And as you can see at the bottom of the screenshot, the classIndex is initialized properly.
But in the second iteration, when loading from DefaultPaths.DEFAULT_NER_MUC_MODEL, it fails:
[image: Bildschirmfoto vom 2019-10-27 23-43-18] https://user-images.githubusercontent.com/19290349/67643095-9457fd00-f913-11e9-94f0-6bf9e8d1d6c2.png Workaround
My current workaround is to just set the ner model to only the threeclass and conll model:
props.setProperty("ner.model", DefaultPaths.DEFAULT_NER_THREECLASS_MODEL + "," + DefaultPaths.DEFAULT_NER_CONLL_MODEL)
But I actually don't what the consequences are if the MUC model is missing.
My theory is that the MUC model is especially large and thus can not be loaded into memory on a mobile device. Is that true? How big is the model in particular?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/stanfordnlp/CoreNLP/issues/961?email_source=notifications&email_token=AA2AYWMEMUCFNNYVLOHSL63QQYEGBA5CNFSM4JFTZIOKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HUUI2KA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA2AYWLVAQOH4CD7645B7ATQQYEGBANCNFSM4JFTZIOA .
erksch
As you can see the same errors as above.
J38
I (somewhat) successfully integrated CoreNLP (3.9.2) in an Android app. The following annotator configuration works just fine:
But as soon as I add the NER annotator I start to get the following error:
The code I use (Kotlin):
The error is very similar to the one described in this issue where the author tried to use the parser annotator.
Debugging
I debugged the stack trace and found that the error is caused by this line (on
classIndex.size()) inCRFClassifier:1480:Meaning
classIndexis null and was not initialized properly.The
classIndexproperty ofCRFClassifieris initialized in theloadClassifier(ObjectInputStream ois, Properties props)method:I found out that the passed ObjectInputStream is effectively a stream on the file from a model path that is determined in the
NERCombinerAnnotatorconstructor:Those
loadPathsare iterated in theloadClassifiersmethod inClassifierCombiner:By adding a breakpoint to this method I found out that the first model path (
DefaultPaths.DEFAULT_NER_THREECLASS_MODEL) in the first iteration of the for-loop is loaded without problems and theclassIndexproperty is set correctly:But in the second iteration, when loading from
DefaultPaths.DEFAULT_NER_MUC_MODEL, it fails:Workaround
My current workaround is to just set the ner model to only the threeclass and conll model:
But I actually don't what the consequences are if the MUC model is missing.
Explanation
My theory is that the MUC model is especially large and thus can not be loaded into memory on a mobile device. Is that true? How big is the model in particular? But when I monitor the memory consumption of the app I don't spot anything critical, the app stays under 512 MB before it crashes.