search

apache / lucene

Apache Lucene open-source search software
https://lucene.apache.org/
Apache License 2.0
2.61k stars 1.02k forks source link

org.apache.lucene.index.IndexFormatTooNewException on arm64 #13452

Open suddendust opened 3 months ago

suddendust commented 3 months ago

Description

I am trying to run Apache Pinot on ARM64 (Graviton). While the table partitions are loaded successfully on x86, I get the following exception on ARM:

org.apache.lucene.index.IndexFormatTooNewException: Format version is not supported (resource ): 9 (needs to be between 6 and 8)
    at org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:213) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
    at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:193) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
    at org.apache.lucene.util.fst.FST.<init>(FST.java:454) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
    at org.apache.pinot.segment.local.segment.index.readers.LuceneFSTIndexReader.<init>(LuceneFSTIndexReader.java:55) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
    at org.apache.pinot.segment.local.segment.index.fst.FstIndexType$ReaderFactory.createIndexReader(FstIndexType.java:200) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
    at org.apache.pinot.segment.local.segment.index.fst.FstIndexType$ReaderFactory.createIndexReader(FstIndexType.java:208) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
    at org.apache.pinot.segment.local.segment.index.fst.FstIndexType$ReaderFactory.createIndexReader(FstIndexType.java:183) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
    at org.apache.pinot.segment.spi.index.IndexReaderFactory$Default.createIndexReader(IndexReaderFactory.java:67) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
    at org.apache.pinot.segment.local.segment.index.column.PhysicalColumnIndexContainer.<init>(PhysicalColumnIndexContainer.java:59) ~[startree-pinot-all-1.2.0-ST.10-jar-with-dependencies.jar:1.2.0-ST.10-ed1c4f0f376a258e77fee8db54706ded71003843]
    at

While I am still in the process of debugging this, I created this issue in case I can get some help faster. Thanks!

Version and environment details

Linux/arm64 Lucene version: 9.10.0

easyice commented 3 months ago

It looks like you are using a lower version of lucene to read the index generated by a higher version. Are you sure the version with this exception is lucene 9.10?

thomasli9895 commented 1 month ago

I also encountered this problem, I upgraded from version 7.8.1 to version 7.17.9 and then fell back to version 7.8.1 and this glitch occurred image

thomasli9895 commented 1 month ago

I also encountered this problem, I upgraded from version 7.8.1 to version 7.17.9 and then fell back to version 7.8.1 and this glitch occurred

org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: IndexFormatTooNewException[Format version is not supported (resource BufferedChecksumIndexInput(SimpleFSIndex Input(path="/usr/share/elasticsearch/workspace/data/nodes/0/_state/segments_41"))): 10 (needs to be between 7 and 9)]; at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.8.1.jar:7.8.1] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.8.1.jar:7.8.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.8.1.jar:7.8.1]