search

apache / lucene

Apache Lucene open-source search software
https://lucene.apache.org/
Apache License 2.0
2.67k stars 1.04k forks source link

shenandoah GC can cause "ClassCastException: org.apache.lucene.codecs.MappingMultiDocsEnum cannot be cast to org.apache.lucene.index.DocsAndPositionsEnum" [LUCENE-5899] #6961

Open asfimport opened 10 years ago

asfimport commented 10 years ago

User report of bizare ClassCastException when running some lucene code with the (experimental) shenandoah GC

Exception in thread "Lucene Merge Thread #0" org.apache.lucene.index.MergePolicy$MergeException: java.lang.ClassCastException: org.apache.lucene.codecs.MappingMultiDocsEnum cannot be cast to org.apache.lucene.index.DocsAndPositionsEnum
    at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.lang.ClassCastException: org.apache.lucene.codecs.MappingMultiDocsEnum cannot be cast to org.apache.lucene.index.DocsAndPositionsEnum
    at org.apache.lucene.codecs.PostingsConsumer.merge(PostingsConsumer.java:127)
    at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:110)
    at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
    at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:399)
    at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:112)
    at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4163)
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3759)
    at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
Exception in thread "Lucene Merge Thread #0" org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: Invalid vLong detected (negative values disallowed)
    at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.io.IOException: Invalid vLong detected (negative values disallowed)
    at org.apache.lucene.store.DataInput.readVLong(DataInput.java:193)
    at org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.<init>(BlockTreeTermsReader.java:169)
    at org.apache.lucene.codecs.lucene41.Lucene41PostingsFormat.fieldsProducer(Lucene41PostingsFormat.java:441)
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:197)
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:254)
    at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:120)
    at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:107)
    at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:143)
    at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:668)
    at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4099)
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3759)
    at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
Exception in thread "Lucene Merge Thread #0" org.apache.lucene.index.MergePolicy$MergeException: java.lang.NullPointerException
    at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.lang.NullPointerException
    at org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.<init>(BlockTreeTermsReader.java:169)
    at org.apache.lucene.codecs.lucene41.Lucene41PostingsFormat.fieldsProducer(Lucene41PostingsFormat.java:441)
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:197)
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:254)
    at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:120)
    at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:107)
    at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:143)
    at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:668)
    at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4099)
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3759)
    at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
Exception in thread "Lucene Merge Thread #0" org.apache.lucene.index.MergePolicy$MergeException: java.lang.NullPointerException
    at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.lang.NullPointerException
    at org.apache.lucene.codecs.MappingMultiDocsAndPositionsEnum.reset(MappingMultiDocsAndPositionsEnum.java:50)
    at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:188)
    at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
    at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:399)
    at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:112)
    at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4163)
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3759)
    at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)

Migrated from LUCENE-5899 by Littlestar, updated Aug 28 2014 Environment:

I test lucene on shenandoah with big memory + bigdata(32g, 128g).   
http://openjdk.java.net/jeps/189
http://icedtea.classpath.org/hg/shenandoah/
asfimport commented 10 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

Hi Littlestar, that's a spooky exception, can you provide a test case showing how you hit it? Or describe what led up to it?

asfimport commented 10 years ago

Littlestar (migrated from JIRA)

I test lucene on shenandoah with big memory + bigdata(128G, 1.2T).
http://openjdk.java.net/jeps/189 http://icedtea.classpath.org/hg/shenandoah/

It throws above exception. I see the luence code, I think the exception is ok. Because org.apache.lucene.codecs.MappingMultiDocsEnum is not instanceof org.apache.lucene.index.DocsAndPositionsEnum.

asfimport commented 10 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

If it works with other garbage collectors it is more likely a bug in Shenandoah. The other one, G1GC, also has many bugs and throws crazy exceptions in some cases (like a NullPointerException where nothing can be null).

From my view at the code, the casts are correct, it is just something Hotspot fails to handle correctly.

Please open bug report at Oracle. We have not yet tested Lucene with this GC, because it is not yet part of official JDK9 preview releases.

asfimport commented 10 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

I've updated the summary & description to make it more clear what circumstances this happens in.

Littlestar: it would be very helpful if you could provide some explicit details on what exactly you mean by "test lucene" ... is this test code you wrote? were you running "ant test" from the lucene distribution? ... which test threw this exception? does it reproduce reliably?

if you do open a JDK bug regarding this, please link to this issue, and then post the resulting bug # back here as a comment as well.

asfimport commented 10 years ago

Littlestar (migrated from JIRA)

I post a mail to shenandoah develop maillist MARK here. http://icedtea.classpath.org/pipermail/shenandoah/2014-August/000163.html

Thanks.

asfimport commented 10 years ago

Littlestar (migrated from JIRA)

I test again, there are some strange exception

org.apache.lucene.index.MergePolicy$MergeException: java.lang.NullPointerException org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: Invalid vLong detected (negative values disallowed)

I post at top.