search

DmitryKey / luke

This is mavenised Luke: Lucene Toolbox Project
Apache License 2.0
1.54k stars 352 forks source link

Issue #175: Offered improvement to reconstruction of unstored fields … #177

Closed chris-bamford closed 4 years ago

chris-bamford commented 4 years ago

…with no position information

DmitryKey commented 4 years ago

@chris-bamford hi, I see some conflicts: https://github.com/DmitryKey/luke/pull/177/conflicts can you help me to resolve these?

chris-bamford commented 4 years ago

Hi @DmitryKey, this is very odd indeed. The 'Resolve conflicts' view above looks very similar to, but not identical to the original code! For example, line 158 declares newPe as a 'PostingsEnum' rather than a 'DocsAndPositionsEnum'. I have never heard of a 'PostingsEnum' so I know this isn't the code I originally forked from. Do you have any idea what might be happening?

If you like we could try again from scratch, but you'd need to guide me through all the steps (sorry!).

DmitryKey commented 4 years ago

@chris-bamford Let me restore the branch with reconstruction logic from git tag. Once that is done, we can compare with your changes again.

DmitryKey commented 4 years ago

@chris-bamford I pushed https://github.com/DmitryKey/luke/tree/luke-4.10.4-field-reconstruction branch. Can you create a pull request against it and see if you get the conflicts resolved?

chris-bamford commented 4 years ago

Hi Dmitry,

I clicked your link (https://github.com/DmitryKey/luke/tree/luke-4.10.4-field-reconstruction) and it took me to that page, whereupon I clicked 'New pull request' - is that what you meant? Doing this gives the view shown in the screenshot which wasn't what I was expecting. I have highlighted 2 areas of concern. Please tell me if/what I am doing wrong.

Thanks! Screenshot 2019-12-16 at 11 38 30

I have compared the changes in those 24 files and the vast majority of them have nothing to do with my changes to DocReconstructor.java. Also, my test file IndexTester2.java does not appear in the list.

chris-bamford commented 4 years ago

Maybe we can set up a call to discuss this? My timezone is GMT.

chris-bamford commented 4 years ago

Replaced with pull request https://github.com/DmitryKey/luke/pull/178

chris-bamford commented 4 years ago

Hi Dmitry,

Happy new year. Is there any way I can build the 4.10.4 version with swing to get the better UI?

Thanks again, Chris

DmitryKey commented 4 years ago

Hi Chris, Happy New Year!

Swing luke does not have the functionality for reconstruction of unstored fields. A suitable tab, following the design of Thinlet based luke, would be Documents.

Current process is to send pull requests directly to lucene: https://github.com/apache/lucene-solr/tree/master/lucene/luke

@mocobeta could you help Chris with the right branch etc where a pull request could be made?

mocobeta commented 4 years ago

Hi Swing Luke bundled with Lucene won't support "re-construct" feature. Because this feature is not compatible with the latest Lucene API.

chris-bamford commented 4 years ago

Hi @mocobeta, that’s sad. I don’t know Lucene 5+ very well as we have been stuck in a time warp at my company for some time. But are you saying that there is no way to access the terms and their positions in unstored fields in later versions? What about term vectors? Seems like a step backwards to me.

mocobeta commented 4 years ago

But are you saying that there is no way to access the terms and their positions in unstored fields in later versions? What about term vectors?

You can traverse terms and their positions, offsets, and term vectors for an indexed field (stored or unstored) in Documents tab (with Thinlet or Swing Luke). I do not understand why "re-construction" is needed for such ordinary posting list access.

スクリーンショット 2020-01-07 15 04 03

スクリーンショット 2020-01-07 15 04 43

This is Swing Luke UI, but I am sure that Thinlet Luke already have the very same functionality. What additional functionality do you need?

chris-bamford commented 4 years ago

In our business we often get questions from customers as to why certain queries did not find something. Being able to reconstruct is invaluable especially as several of our key fields are unstored. Our support team uses it frequently. This is why we were keen to fix the bug in the 4.10.4 version. Seeing the terms together in the order they went in, with position gaps helps us understand why (say) a phrase query which spans multiple adjacent terms didn’t match.

mocobeta commented 4 years ago

Seeing the terms together in the order they went in, with position gaps helps us understand why (say) a phrase query which spans multiple adjacent terms didn’t match.

Ok. The latest Swing Luke doesn't have "re-construct" button. I dropped some minor features when porting UI from Thinlet to Swing (to be precise, it's not just porting "UI" but building completely new Luke from scratch). I gave clear priority to "making Luke live longer" with latest Java, instead of re-implementing entire features of Thinlet Luke, whose code was outdated and could not stand with future Java updates. (I am just a volunteer, and it was the best decision I can do as a open source project maintainer.)

If you really need this feature with the latest Luke, you can send a pull request to Apache Lucene project.

chris-bamford commented 4 years ago

@mocobeta I fully understand, I would have done the same. Keeping Luke going for longer has to be the main priority. I will discuss with my team to see if we can find resources to add the feature.

Kind regards Chris

mocobeta commented 4 years ago

@chris-bamford I opened an Lucene issue for this. https://issues.apache.org/jira/browse/LUCENE-9119

I think it's technically possible with recent Lucene APIs (though we cannot directly adopt the same patch here).