search

apache / lucene

Apache Lucene open-source search software
https://lucene.apache.org/
Apache License 2.0
2.59k stars 1.01k forks source link

TestIndexWriterExceptions.testNoLostDeletesOrUpdates() failure [LUCENE-7724] #8775

Open asfimport opened 7 years ago

asfimport commented 7 years ago

Non-reproducing branch_6x seed from my Jenkins:

Checking out Revision bce1417fceeed2054f16565e96dc49268c1b2ea1 (refs/remotes/origin/branch_6x)
[...]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterExceptions -Dtests.method=testNoLostDeletesOrUpdates -Dtests.seed=611686EB379EF662 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=en-US -Dtests.timezone=America/Cordoba -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   9.90s J6  | TestIndexWriterExceptions.testNoLostDeletesOrUpdates <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=343, name=Lucene Merge Thread #21, state=RUNNABLE, group=TGRP-TestIndexWriterExceptions]
   [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
   [junit4]    >    at __randomizedtesting.SeedInfo.seed([611686EB379EF662]:0)
   [junit4]    >    at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
   [junit4]    >    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
   [junit4]    > Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
   [junit4]    >    at org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:345)
   [junit4]    >    at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:696)
   [junit4]    >    at org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:691)
   [junit4]    >    at org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:4964)
   [junit4]    >    at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4455)
   [junit4]    >    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3940)
   [junit4]    >    at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
   [junit4]    >    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
   [junit4]    > Caused by: org.apache.lucene.store.MockDirectoryWrapper$FakeIOException
   [junit4]    >    at org.apache.lucene.index.TestIndexWriterExceptions$11.eval(TestIndexWriterExceptions.java:1875)
   [junit4]    >    at org.apache.lucene.store.MockDirectoryWrapper.maybeThrowDeterministicException(MockDirectoryWrapper.java:1022)
   [junit4]    >    at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:144)
   [junit4]    >    at org.apache.lucene.store.MockIndexOutputWrapper.writeByte(MockIndexOutputWrapper.java:126)
   [junit4]    >    at org.apache.lucene.store.DataOutput.writeInt(DataOutput.java:70)
   [junit4]    >    at org.apache.lucene.codecs.CodecUtil.writeHeader(CodecUtil.java:90)
   [junit4]    >    at org.apache.lucene.codecs.CodecUtil.writeIndexHeader(CodecUtil.java:133)
   [junit4]    >    at org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.<init>(Lucene54DocValuesConsumer.java:74)
   [junit4]    >    at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsConsumer(Lucene54DocValuesFormat.java:108)
   [junit4]    >    at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:213)
   [junit4]    >    at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:111)
   [junit4]    >    at org.apache.lucene.index.ReadersAndUpdates.handleNumericDVUpdates(ReadersAndUpdates.java:328)
   [junit4]    >    at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:521)
   [junit4]    >    at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:272)
   [junit4]    >    at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:4119)
   [junit4]    >    at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:4077)
   [junit4]    >    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
   [junit4]    >    ... 2 moreThrowable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=344, name=Lucene Merge Thread #22, state=RUNNABLE, group=TGRP-TestIndexWriterExceptions]
   [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.store.MockDirectoryWrapper$FakeIOException
   [junit4]    >    at __randomizedtesting.SeedInfo.seed([611686EB379EF662]:0)
   [junit4]    >    at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
   [junit4]    >    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
   [junit4]    > Caused by: org.apache.lucene.store.MockDirectoryWrapper$FakeIOException
   [junit4]    >    at org.apache.lucene.index.TestIndexWriterExceptions$11.eval(TestIndexWriterExceptions.java:1875)
   [junit4]    >    at org.apache.lucene.store.MockDirectoryWrapper.maybeThrowDeterministicException(MockDirectoryWrapper.java:1022)
   [junit4]    >    at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:144)
   [junit4]    >    at org.apache.lucene.store.MockIndexOutputWrapper.writeByte(MockIndexOutputWrapper.java:126)
   [junit4]    >    at org.apache.lucene.store.DataOutput.writeInt(DataOutput.java:70)
   [junit4]    >    at org.apache.lucene.codecs.CodecUtil.writeHeader(CodecUtil.java:90)
   [junit4]    >    at org.apache.lucene.codecs.CodecUtil.writeIndexHeader(CodecUtil.java:133)
   [junit4]    >    at org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.<init>(Lucene54DocValuesConsumer.java:74)
   [junit4]    >    at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsConsumer(Lucene54DocValuesFormat.java:108)
   [junit4]    >    at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:213)
   [junit4]    >    at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:111)
   [junit4]    >    at org.apache.lucene.index.ReadersAndUpdates.handleNumericDVUpdates(ReadersAndUpdates.java:328)
   [junit4]    >    at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:521)
   [junit4]    >    at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:272)
   [junit4]    >    at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:4119)
   [junit4]    >    at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:4077)
   [junit4]    >    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
   [junit4]    >    at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
   [junit4]    >    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=false,coord=no): {content6=DFR I(F)BZ(0.3), field=DFR I(n)BZ(0.3), content4=DFR I(F)L1, contents=DFR I(n)1, content2=DFR I(ne)Z(0.3), content1=IB LL-L1, id=DFR I(n)L3(800.0), content=DFR I(F)1}, locale=en-US, timezone=America/Cordoba
   [junit4]   2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation 1.8.0_77 (64-bit)/cpus=16,threads=1,free=421632952,total=456654848

Migrated from LUCENE-7724 by Steven Rowe (@sarowe)

asfimport commented 7 years ago

Steven Rowe (@sarowe) (migrated from JIRA)

Beasting with the repro line (after s/test/test-nocompile/ [1]) on the tip of branch_6x failed 3/150 iterations on my box -> 2% failure rate.

[1] ant compile-test ; (for a in \{1..500\} ; do ant test-nocompile <remainder of repro line> ; done) 2>&1 | tee \~/output.txt