search

apache / lucene

Apache Lucene open-source search software
https://lucene.apache.org/
Apache License 2.0
2.67k stars 1.03k forks source link

TestRandomChains.testRandomChains() failure: got startOffset=10,endOffset=9 [LUCENE-7711] #8762

Open asfimport opened 7 years ago

asfimport commented 7 years ago

Found while beasting TestRandomChains for #8759 (note though that the failure below reproduces on a clean master checkout):

   [junit4] Suite: org.apache.lucene.analysis.core.TestRandomChains
   [junit4]   2> TEST FAIL: useCharFilter=false text='\ufac4\u0552H \ua954\ua944 \ud0d2\uaddd\ub6cb\uc388\uc344\uca88\ud224\uc462\uaf42 g '
   [junit4]   2> Exception from random analyzer: 
   [junit4]   2> charfilters=
   [junit4]   2>   org.apache.lucene.analysis.charfilter.HTMLStripCharFilter(java.io.StringReader@72c69dd0, [<HOST>, <HANGUL>, <IDEOGRAPHIC>, <SOUTHEAST_ASIAN>])
   [junit4]   2> tokenizer=
   [junit4]   2>   org.apache.lucene.analysis.standard.StandardTokenizer(org.apache.lucene.util.AttributeFactory$1@2ff87e59)
   [junit4]   2> filters=
   [junit4]   2>   org.apache.lucene.analysis.miscellaneous.KeywordRepeatFilter(ValidatingTokenFilter@2c2ac1cc term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,flags=0,payload=null,keyword=false)
   [junit4]   2>   org.apache.lucene.analysis.cjk.CJKBigramFilter(ValidatingTokenFilter@51f2f8f0 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,flags=0,payload=null,keyword=false)
   [junit4]   2>   org.apache.lucene.analysis.shingle.ShingleFilter(ValidatingTokenFilter@2ea3d3ba term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,flags=0,payload=null,keyword=false, 49)
   [junit4]   2>   org.apache.lucene.analysis.in.IndicNormalizationFilter(ValidatingTokenFilter@25d58ed7 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,flags=0,payload=null,keyword=false)
   [junit4]   2> offsetsAreCorrect=false
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestRandomChains -Dtests.method=testRandomChains -Dtests.seed=E532502212098AC7 -Dtests.slow=true -Dtests.locale=ko-KR -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.23s | TestRandomChains.testRandomChains <<<
   [junit4]    > Throwable #1: java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset; got startOffset=10,endOffset=9
   [junit4]    >    at __randomizedtesting.SeedInfo.seed([E532502212098AC7:D8D37943551B9707]:0)
   [junit4]    >    at org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl.setOffset(PackedTokenAttributeImpl.java:110)
   [junit4]    >    at org.apache.lucene.analysis.shingle.ShingleFilter.incrementToken(ShingleFilter.java:345)
   [junit4]    >    at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:67)
   [junit4]    >    at org.apache.lucene.analysis.in.IndicNormalizationFilter.incrementToken(IndicNormalizationFilter.java:40)
   [junit4]    >    at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:67)
   [junit4]    >    at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:731)
   [junit4]    >    at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:642)
   [junit4]    >    at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:540)
   [junit4]    >    at org.apache.lucene.analysis.core.TestRandomChains.testRandomChains(TestRandomChains.java:853)
   [junit4]    >    at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {dummy=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{}, maxPointsInLeafNode=542, maxMBSortInHeap=7.773738401752009, sim=RandomSimilarity(queryNorm=false): {}, locale=ko-KR, timezone=Atlantic/Jan_Mayen
   [junit4]   2> NOTE: Mac OS X 10.12.3 x86_64/Oracle Corporation 1.8.0_112 (64-bit)/cpus=8,threads=1,free=225448184,total=257425408
   [junit4]   2> NOTE: All tests run in this JVM: [TestRandomChains]
   [junit4] Completed [1/1 (1!)] in 1.92s, 1 test, 1 error <<< FAILURES!

Migrated from LUCENE-7711 by Steven Rowe (@sarowe), updated Jul 10 2017

asfimport commented 7 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

Another reproducing failure on branch_7_0, from my Jenkins:

Checking out Revision 6cdc0060e5c3b93f0764d7e8e441fa21931fe60d (refs/remotes/origin/branch_7_0)
[...]
   [junit4] Suite: org.apache.lucene.analysis.core.TestRandomChains
   [junit4]   2> TEST FAIL: useCharFilter=true text='arr]] [[ja:\u30de\u30b7\u30e5\u30fc\u30fb\u30d1\u30fc\u30eb '
   [junit4]   2> Exception from random analyzer: 
   [junit4]   2> charfilters=
   [junit4]   2> tokenizer=
   [junit4]   2>   org.apache.lucene.analysis.standard.StandardTokenizer()
   [junit4]   2> filters=
   [junit4]   2>   org.apache.lucene.analysis.miscellaneous.KeywordRepeatFilter(ValidatingTokenFilter@218bc4e4 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,keyword=false)
   [junit4]   2>   org.apache.lucene.analysis.core.DecimalDigitFilter(ValidatingTokenFilter@48b9b511 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,keyword=false)
   [junit4]   2>   org.apache.lucene.analysis.miscellaneous.HyphenatedWordsFilter(ValidatingTokenFilter@6800dfe4 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,keyword=false)
   [junit4]   2>   org.apache.lucene.analysis.cjk.CJKBigramFilter(ValidatingTokenFilter@507e81ab term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,keyword=false)
   [junit4]   2> offsetsAreCorrect=false
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestRandomChains -Dtests.method=testRandomChains -Dtests.seed=2577EBB6844BD489 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=en-ZA -Dtests.timezone=America/Argentina/Jujuy -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   18.9s J2 | TestRandomChains.testRandomChains <<<
   [junit4]    > Throwable #1: java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, and offsets must not go backwards startOffset=11,endOffset=13,lastStartOffset=13 for field 'dummy'
   [junit4]    >    at __randomizedtesting.SeedInfo.seed([2577EBB6844BD489:1896C2D7C359C949]:0)
   [junit4]    >    at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:769)
   [junit4]    >    at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:431)
   [junit4]    >    at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:393)
   [junit4]    >    at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)
   [junit4]    >    at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
   [junit4]    >    at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1717)
   [junit4]    >    at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1462)
   [junit4]    >    at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:171)
   [junit4]    >    at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:650)
   [junit4]    >    at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:540)
   [junit4]    >    at org.apache.lucene.analysis.core.TestRandomChains.testRandomChains(TestRandomChains.java:856)