search

DmitryKey / luke

This is mavenised Luke: Lucene Toolbox Project
Apache License 2.0
1.54k stars 352 forks source link

Luke cannot open ES index file with completion suggestion #152

Closed begemot63 closed 5 years ago

begemot63 commented 5 years ago

If I add completion suggestion to the ES mapping then Luke cannot open the index file. Assume I add the following suggestion in ES mapping: { "suggestible_text_fields": { "match": "*_st_en", "mapping": { "type": "text", "analyzer": "custom_english_index_analyzer", "search_analyzer": "custom_english_query_analyzer", "fields": { "bigrammed": { "type": "text", "analyzer": "custom_english_bigram_index_analyzer" }, "exact": { "type": "text", "analyzer": "custom_exact_analyzer" }, "suggest": { "type": "completion" } } } } Luke throws exception : [2018-12-06 17:48:32,456] ERROR (IndexHandler.java:62) - An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'completion' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [Lucene50, IDVersion] java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'completion' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [Lucene50, IDVersion] at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:116) at org.apache.lucene.codecs.PostingsFormat.forName(PostingsFormat.java:112) at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.(PerFieldPostingsFormat.java:289) at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372) at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:113) at org.apache.lucene.index.SegmentReader.(SegmentReader.java:83) at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:66) at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:688) at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:81) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63) at org.apache.lucene.luke.util.IndexUtils$1.preVisitDirectory(IndexUtils.java:104) at org.apache.lucene.luke.util.IndexUtils$1.preVisitDirectory(IndexUtils.java:99) at java.nio.file.Files.walkFileTree(Files.java:2677) at java.nio.file.Files.walkFileTree(Files.java:2742) at org.apache.lucene.luke.util.IndexUtils.openIndex(IndexUtils.java:99) at org.apache.lucene.luke.app.IndexHandler.open(IndexHandler.java:60) at org.apache.lucene.luke.app.desktop.components.dialog.menubar.OpenIndexDialogFactoryImpl$ListenerFunctions.openIndexOrDirectory(OpenIndexDialogFactoryImpl.java:316) at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022) at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348) at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402) at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259) at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252) at java.awt.Component.processMouseEvent(Component.java:6535) at javax.swing.JComponent.processMouseEvent(JComponent.java:3324) at java.awt.Component.processEvent(Component.java:6300) at java.awt.Container.processEvent(Container.java:2236) at java.awt.Component.dispatchEventImpl(Component.java:4891) at java.awt.Container.dispatchEventImpl(Container.java:2294) at java.awt.Component.dispatchEvent(Component.java:4713) at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4888) at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4525) at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4466) at java.awt.Container.dispatchEventImpl(Container.java:2280) at java.awt.Window.dispatchEventImpl(Window.java:2750) at java.awt.Component.dispatchEvent(Component.java:4713) at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758) at java.awt.EventQueue.access$500(EventQueue.java:97) at java.awt.EventQueue$3.run(EventQueue.java:709) at java.awt.EventQueue$3.run(EventQueue.java:703) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86) at java.awt.EventQueue$4.run(EventQueue.java:731) at java.awt.EventQueue$4.run(EventQueue.java:729) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76) at java.awt.EventQueue.dispatchEvent(EventQueue.java:728) at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:109) at java.awt.WaitDispatchSupport$2.run(WaitDispatchSupport.java:184) at java.awt.WaitDispatchSupport$4.run(WaitDispatchSupport.java:229) at java.awt.WaitDispatchSupport$4.run(WaitDispatchSupport.java:227) at java.security.AccessController.doPrivileged(Native Method) at java.awt.WaitDispatchSupport.enter(WaitDispatchSupport.java:227) at java.awt.Dialog.show(Dialog.java:1084) at java.awt.Component.show(Component.java:1673) at java.awt.Component.setVisible(Component.java:1625) at java.awt.Window.setVisible(Window.java:1014) at java.awt.Dialog.setVisible(Dialog.java:1005) at org.apache.lucene.luke.app.desktop.util.DialogOpener.open(DialogOpener.java:43) at org.apache.lucene.luke.app.desktop.util.DialogOpener.open(DialogOpener.java:36) at org.apache.lucene.luke.app.desktop.components.MenuBarProvider$ListenerFunctions.showOpenIndexDialog(MenuBarProvider.java:185) at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022) at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348) at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402) at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259) at javax.swing.AbstractButton.doClick(AbstractButton.java:376) at javax.swing.plaf.basic.BasicMenuItemUI.doClick(BasicMenuItemUI.java:833) at com.apple.laf.AquaMenuItemUI.doClick(AquaMenuItemUI.java:157) at javax.swing.plaf.basic.BasicMenuItemUI$Handler.mouseReleased(BasicMenuItemUI.java:877) at java.awt.Component.processMouseEvent(Component.java:6535) at javax.swing.JComponent.processMouseEvent(JComponent.java:3324) at java.awt.Component.processEvent(Component.java:6300) at java.awt.Container.processEvent(Container.java:2236) at java.awt.Component.dispatchEventImpl(Component.java:4891) at java.awt.Container.dispatchEventImpl(Container.java:2294) at java.awt.Component.dispatchEvent(Component.java:4713) at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4888) at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4525) at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4466) at java.awt.Container.dispatchEventImpl(Container.java:2280) at java.awt.Window.dispatchEventImpl(Window.java:2750) at java.awt.Component.dispatchEvent(Component.java:4713) at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758) at java.awt.EventQueue.access$500(EventQueue.java:97) at java.awt.EventQueue$3.run(EventQueue.java:709) at java.awt.EventQueue$3.run(EventQueue.java:703) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86) at java.awt.EventQueue$4.run(EventQueue.java:731) at java.awt.EventQueue$4.run(EventQueue.java:729) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76) at java.awt.EventQueue.dispatchEvent(EventQueue.java:728) at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116) at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93) at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

Note, completion suggestion works in ES very well, but Luke cannot open index file. I am using ES 6.4.2

mocobeta commented 5 years ago

Thanks for sharing the stack trace. FYI, this seems to be related to #129

begemot63 commented 5 years ago

I did not find "Load custom codecs" button to try #129

mocobeta commented 5 years ago

129 is not resolved yet - so we need a patch for it. I will work for this when I can take time for it.

It seems that elasticsearch uses their own custom codecs for the auto completion fields. I don't like to include elasticsearch jars or classes in Luke distribution, so we should support "loading custom codecs" feature to add the codecs on runtime.

2018-12-06 17:48:32,456] ERROR (IndexHandler.java:62) - An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'completion' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [Lucene50, IDVersion]

For quick fix, you may be able to add elasticsearch jars to the classpath when launching Luke by modifying luke.sh. Though I have not tried yet that.

begemot63 commented 5 years ago

I added commons-codec-1.10.jar to the classpath in luke.sh but still see the same error:

[2018-12-09 15:06:52,343] ERROR (IndexHandler.java:62) - An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'completion' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [Lucene50, IDVersion] java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'completion' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [Lucene50, IDVersion] at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:116)

Please let me know what jar needs to be included into classpath to handle it?

mocobeta commented 5 years ago

Elasticsearch should include the custom codec class, so you need the elasticsearch jars you are using when launching Luke. I have not tried yet, but I'd add all jars (except for lucene jars, which are already included in Luke) in the elasticsearch lib directory.

begemot63 commented 5 years ago

I added commons-codec-1.10.jar, elasticsearch-6.4.2.jar, elasticsearch-core-6.4.2.jar, elasticsearch-x-content-6.4.2.jar to the classpath and it does not help

mocobeta commented 5 years ago

Sorry, I found that the missed codec class is not elasticsearch's one. Lucene's suggester module includes the custom codec named 'completion' and Luke 7.5.0 does not include this jar. I will add this to Luke distribution as of the next release. Thanks for the report. https://mvnrepository.com/artifact/org.apache.lucene/lucene-suggest

For now, can you try to add the jar to the classpath?

mocobeta commented 5 years ago

I just added lucene-suggest to the pom.xml dependencies in the master branch. You can checkout and try a nightly build by mvn package.

begemot63 commented 5 years ago

I confirm. The issue is solved now. Thanks

mocobeta commented 5 years ago

Thanks for the confirmation!

begemot63 commented 5 years ago

One second. Luke can open the index but show result only if ElasticSearch is down . That seen with suggestion field. Please review it

begemot63 commented 5 years ago

let me know if we need to open a new issue

mocobeta commented 5 years ago

Luke can open the index but show result only if ElasticSearch is down . That seen with suggestion field.

I'm not sure, but Elasticsearch could have the index in RAM only in a while to avoid disk I/O (and dump it to disk later) - in that case Luke cannot read the indexed field until it is flushed to the disk and it's not a bug. Is the field persisted when opening it by Luke?