Add a Lucene 9.8 engine.

[jpountz]

This PR does a few changes:

• Removes the lucene-9.6.0 engine which is actually built on 8.10.1 rather than 9.6.0.
• Adds a lucene-9.8.0 engine.
• Reuses the original Lucene JARs instead of creating an uber JAR.
• Makes indexing multi-threaded so that it takes less time to build the index (just for convenience since indexing is not measured).
• Adds document reordering to speed up query processing, especially conjunctions.
• TOP_10 no longer counts up to 1,000 hits, it stops at 10.

Closes #44

[jpountz]

Lucene 9.8 is released, this PR is now ready from my point of view.

[PSeitz]

Thanks! How does the index reordering work?

[jpountz]

This implements an algorithm called "Recursive graph bisection", which is the same algorithm that the pisa-0.8.2 engine uses. Here is the Lucene PR where it was implemented: https://github.com/apache/lucene/pull/12489.

This algorithm tries to reorder doc IDs to minimize the sum of the logs of gaps between consecutive doc IDs in postings. This helps make postings more compact, but this also has the interesting side-effect of clustering documents that have similar postings together, which in-turn helps conjunctions.

The original paper is good but I found this follow-up reproducibility study to be a better introduction to this algorithm.

[PSeitz]

That's pretty interesting. I think it would be nice to make this configurable

[jpountz]

Do you have opinions on how to make this configurable? FWIW testing performance without reordering is as simple as replacing idx-reordered with idx in the Makefile target that runs queries. If we want to make it easier to test with and without doc ID reordering, I think I could easily split the Lucene 9.8 engine into lucene-9.8 and lucene-9.8-bp or something like that.

[PSeitz]

For now I think it would be good to have lucene-9.8 and lucene-9.8-reorder

[jpountz]

@PSeitz It's now split, I used the name lucene-9.8.0-bp if you don't mind to be more explicit about what reordering is implemented (e.g. another option is to sort the index by a min_hash of the terms of each document).

I'm not too familiar with make so there might be better ways of doing what I tried to do...

[PSeitz]

Thanks! There may be better ways with make, but the duplication variant is unamibigious