High concurrency, stanfordnlp out of memory error.

[Paper-Heart]

Current Circumstances: I've integrated the stanford ner functionality into a java service that has tens of millions of calls per day. I deployed in 3 clusters and allocated 10GB of memory per pod.

Issue: Over a period of about seven days, Pods not available and continued to report OutOfMemory Exceptions.

Calling Code:

    private static final Log LOG = LogFactory.getInstance(StanfordNlpApp.class);

    private static StanfordCoreNLP pipeline;

    //executed once on system start
    public static void initNLP() {
        pipeline = new StanfordCoreNLP("stanford-hanweb-chinese.properties");
    }

    public static List<NerEntityBean> nerByText(String text) {
        List<NerEntityBean> nerEntityBeans = new ArrayList<>();
        long startTime = System.currentTimeMillis();
        Annotation document = new Annotation(text);
        pipeline.annotate(document);
        List<CoreMap> sentences = document.get(CoreAnnotations.SentencesAnnotation.class);
        for (CoreMap sentence : sentences) {
            for (CoreLabel token : sentence.get(CoreAnnotations.TokensAnnotation.class)) {
                String ner = token.get(CoreAnnotations.NamedEntityTagAnnotation.class);
                int pos = token.beginPosition();
                if (StringUtil.equals(ner, "O")) {
                    continue;
                }
                String word = token.get(CoreAnnotations.TextAnnotation.class);
                NerEntityBean nerEntityBean = new NerEntityBean();
                nerEntityBean.setPos(pos);
                nerEntityBean.setWord(word);
                nerEntityBean.setType(ner);
                nerEntityBeans.add(nerEntityBean);
            }
        }
        long endTime = System.currentTimeMillis();
        long costTime = endTime - startTime;
        LOG.debug("StanfordNlp cost : " + costTime);
        return nerEntityBeans;
    }

Configuration File:

# Pipeline options - lemma is no-op for Chinese but currently needed because coref demands it (bad old requirements system)
annotators = tokenize, ssplit, pos, lemma, ner, parse, coref

# segment
tokenize.language = zh
segment.model = edu/stanford/nlp/models/segmenter/chinese/ctb.gz
segment.sighanCorporaDict = edu/stanford/nlp/models/segmenter/chinese
segment.serDictionary = edu/stanford/nlp/models/segmenter/chinese/dict-chris6.ser.gz,data/model/name-dict.ser.gz
segment.sighanPostProcessing = true

# sentence split
ssplit.boundaryTokenRegex = [.?]|[!???]+

# pos
pos.model = edu/stanford/nlp/models/pos-tagger/chinese-distsim.tagger

# ner
ner.language = chinese
ner.model = edu/stanford/nlp/models/ner/chinese.misc.distsim.crf.ser.gz
ner.applyNumericClassifiers = true
ner.useSUTime = false

# regexner
ner.fine.regexner.mapping = edu/stanford/nlp/models/kbp/chinese/gazetteers/cn_regexner_mapping.tab
ner.fine.regexner.noDefaultOverwriteLabels = CITY,COUNTRY,STATE_OR_PROVINCE

# parse
parse.model = edu/stanford/nlp/models/srparser/chineseSR.ser.gz

# depparse
depparse.model    = edu/stanford/nlp/models/parser/nndep/UD_Chinese.gz
depparse.language = chinese

# coref
coref.sieves = ChineseHeadMatch, ExactStringMatch, PreciseConstructs, StrictHeadMatch1, StrictHeadMatch2, StrictHeadMatch3, StrictHeadMatch4, PronounMatch
coref.input.type = raw
coref.postprocessing = true
coref.calculateFeatureImportance = false
coref.useConstituencyTree = true
coref.useSemantics = false
coref.algorithm = hybrid
coref.path.word2vec =
coref.language = zh
coref.defaultPronounAgreement = true
coref.zh.dict = edu/stanford/nlp/models/dcoref/zh-attributes.txt.gz
coref.print.md.log = false
coref.md.type = RULE
coref.md.liberalChineseMD = false

# kbp
kbp.semgrex = edu/stanford/nlp/models/kbp/chinese/semgrex
kbp.tokensregex = edu/stanford/nlp/models/kbp/chinese/tokensregex
kbp.language = zh
kbp.model = none

# entitylink
entitylink.wikidict = edu/stanford/nlp/models/kbp/chinese/wikidict_chinese.tsv.gz

Error Log:

Call method should have no problem. In theory, memory should be cleaned by the GC and no more than 10GB of memory should be consumed. Can help solve the out of memory problem.Thanks a lot.

[AngledLuffa]

Is there any way you can get us a snapshot of the heap profile when it runs out of memory like that? Especially a profile of its allocation traces would be helpful. We have a demo server that stops working every couple months, and I believe the reason is it has too much memory fragmentation or otherwise runs out of memory, but with how rarely it happens it's hard to debug.

[Paper-Heart]

We use java services to provide functionality, do you mean to print out the jstack information and send it to you after we serve out of memory?

[AngledLuffa]

Well, a heap profile would be better, if that's in any way possible. The last time I got a stack trace from my server, the threads were all stuck waiting on allocations, as far as I could tell, but I don't think I actually got any actionable information from that attempt.

[Paper-Heart]

Is the heap profile a tool that is used to monitor the heap, or something else.I haven't used this tool before, perhaps you could give me a brief introduction on how to use it to help you deal with this problem.

Today, I looked at the error log of the service and found a new problem causing stackoverflow. It's not a common occurrence, and it only happened three times during the day. The attached txt file is the log information. error_log.txt

This kind of error, whether because some special circumstances cause the code infinite recursion.I tried several scenarios and performed pressure tests in a local environment and did not reproduce the problem. Hope to get your reply,thanks.

[AngledLuffa]

That other error must have been a very unusual sentence. I can change it so there isn't a recursive call, but rather it iterates over the nodes in a loop.

As for the heap profile, it looks like some of the tools I used to use are no longer shipped with recent versions of Java. I'll have to investigate another way of debugging the current heap state of the server.