search

morfologik / morfologik-stemming

Tools for finite state automata construction and dictionary-based morphological dictionaries. Includes Polish stemming dictionary.
BSD 3-Clause "New" or "Revised" License
186 stars 44 forks source link

InvalidMarkExceptions thrown from BufferUtils #89

Closed nbelisle11 closed 7 years ago

nbelisle11 commented 7 years ago

I tried to dig into this one but I can't figure out why this is happening based on source code analysis nor can I reliably reproduce it. It seems to be happening intermittently. Error & full stacktrace: java.lang.RuntimeException: Could not check sentence (language: English (US)): 'mixture'

java.nio.InvalidMarkException: null
    at java.nio.Buffer.reset(Buffer.java:306)
    at morfologik.stemming.BufferUtils.charsToBytes(BufferUtils.java:177)
    at morfologik.speller.Speller.charSequenceToBytes(Speller.java:195)
    at morfologik.speller.Speller.isInDictionary(Speller.java:259)
    at morfologik.speller.Speller.isMisspelled(Speller.java:239)
    at org.languagetool.rules.spelling.morfologik.MorfologikSpeller.isMisspelled(MorfologikSpeller.java:86)
    at org.languagetool.rules.spelling.morfologik.MorfologikMultiSpeller.isMisspelled(MorfologikMultiSpeller.java:129)
    at org.languagetool.rules.spelling.morfologik.MorfologikSpellerRule.isMisspelled(MorfologikSpellerRule.java:157)
    at org.languagetool.rules.spelling.morfologik.MorfologikSpellerRule.getRuleMatches(MorfologikSpellerRule.java:178)
    at org.languagetool.rules.en.AbstractEnglishSpellerRule.getRuleMatches(AbstractEnglishSpellerRule.java:45)
    at org.languagetool.rules.spelling.morfologik.MorfologikSpellerRule.match(MorfologikSpellerRule.java:107)
    at org.languagetool.JLanguageTool.checkAnalyzedSentence(JLanguageTool.java:644)
    at org.languagetool.JLanguageTool$TextCheckCallable.call(JLanguageTool.java:991)
    at org.languagetool.JLanguageTool$TextCheckCallable.call(JLanguageTool.java:943)
    at org.languagetool.JLanguageTool.performCheck(JLanguageTool.java:606)
    at org.languagetool.JLanguageTool.check(JLanguageTool.java:558)
    at org.languagetool.JLanguageTool.check(JLanguageTool.java:508)
    at org.languagetool.JLanguageTool.check(JLanguageTool.java:504)

Might be related in some way to https://github.com/morfologik/morfologik-stemming/issues/75 and/or https://github.com/morfologik/morfologik-stemming/issues/90

nbelisle11 commented 7 years ago

In this example https://github.com/morfologik/morfologik-stemming/blob/master/morfologik-speller/src/main/java/morfologik/speller/Speller.java#L227 is being called with String param mixture

dweiss commented 7 years ago

Make sure speller is not used in a multi-threaded environment. Looks like you're calling it from multiple threads concurrently and it messes up internal buffers.

dweiss commented 7 years ago

Let me know if this was indeed the case.

nbelisle11 commented 7 years ago

We are indeed using this in a multithreaded environment. I'll adjust and reopen if this doesn't resolve the issue. Thanks for the prompt response!