search

apache / lucene

Apache Lucene open-source search software
https://lucene.apache.org/
Apache License 2.0
2.69k stars 1.04k forks source link

Elias-Fano sequence also on BytesRef [LUCENE-5524] #6587

Closed asfimport closed 9 years ago

asfimport commented 10 years ago

Migrated from LUCENE-5524 by Paul Elschot, 1 vote, resolved May 19 2015 Attachments: LUCENE-5524-20141126.patch

asfimport commented 10 years ago

ASF GitHub Bot (migrated from JIRA)

GitHub user PaulElschot opened a pull request:

https://github.com/apache/lucene-solr/pull/41

Elias-Fano sequence also on BytesRef

LUCENE-5524

This PR adds encoding/decoding an Elias-Fano sequence in/from a BytesRef.
This PR adds three classes:
EliasFanoLongs and EliasFanoBytes both extending EliasFanoSequence,
and the long[] encoding is moved from EliasFanoEncoder into EliasFanoLongs.
The EliasFanoDecoder is changed to use these classes.
(There are also some improved variable names, this makes the changes somewhat less easy to read...)

The recent fix for the number of index entry bits is included.

This PR also adds methods readVlong and writeVLong to BytesRef. I considered keeping them local in EliasFanoBytes, but these fit better in BytesRef I think.

This PR also changes EliasFanoDocIdSet to use EliasFanoLongs, and to fall back to a FixedBitSet when too many bits are set. This fall back could be a separate issue, but that would be more work now.

I hope I got the generics and diamonds right...
This is a squashed commit against trunk, 12 March 2014

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/PaulElschot/lucene-solr efbytesref-201403a

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/41.patch

To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message:

This closes `#41`

commit 4c302a5b8b03ff96dc486cbffd0163c28707a549 Author: Paul Elschot <paul.j.elschot@gmail.com> Date: 2014-03-12T21:21:13Z

Squashed commit of efbytesref, 20140312
asfimport commented 10 years ago

ASF GitHub Bot (migrated from JIRA)

GitHub user PaulElschot opened a pull request:

https://github.com/apache/lucene-solr/pull/45

Squashed commit of efbytesref, 20140422

LUCENE-5524

This PR adds encoding/decoding an Elias-Fano sequence in/from a BytesRef.
This PR adds three classes:
EliasFanoLongs and EliasFanoBytes both extending EliasFanoSequence,
and the long[] encoding is moved from EliasFanoEncoder into EliasFanoLongs.
The EliasFanoDecoder is changed to use these classes.
(There are also some improved variable names, this makes the changes somewhat less easy to read...)

The recent fix for the number of index entry bits is included.

This PR also adds methods readVlong and writeVLong to BytesRef. I considered keeping them local in EliasFanoBytes, but these fit better in BytesRef I think.

This PR also changes EliasFanoDocIdSet to use EliasFanoLongs, and to fall back to a FixedBitSet when too many bits are set. This fall back could be a separate issue, but that would be more work now.

I hope I got the generics and diamonds right...
This is a squashed commit against trunk, 12 March 2014

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/PaulElschot/lucene-solr efbytesref-201404a

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/45.patch

To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message:

This closes `#45`

commit a214913ac2143277d9539a3e9e3d1cd1662b754a Author: Paul Elschot <paul.j.elschot@gmail.com> Date: 2014-03-12T21:21:13Z

Squashed commit of efbytesref, 20140312
asfimport commented 10 years ago

ASF GitHub Bot (migrated from JIRA)

Github user PaulElschot closed the pull request at:

https://github.com/apache/lucene-solr/pull/41
asfimport commented 10 years ago

Paul Elschot (migrated from JIRA)

PR 45 is the same as PR 41, rebased to trunk of 22 April 2014.

asfimport commented 10 years ago

ASF GitHub Bot (migrated from JIRA)

GitHub user PaulElschot opened a pull request:

https://github.com/apache/lucene-solr/pull/50

efbytesref 201405a1

LUCENE-5524

This closes `#45`

Update to recent trunk, small javadoc correction, use readVLong and writeVLong from ByteArrayDataInput/Output instead of from BytesRef.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/PaulElschot/lucene-solr efbytesref-201405a1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/50.patch

To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message:

This closes `#50`

commit d35d6ceb669d30dc169dd3e2fbb993833e2c1a82 Author: Paul Elschot <paul.j.elschot@gmail.com> Date: 2014-05-13T16:13:24Z

efbytesref 201405a1
asfimport commented 10 years ago

Mikhail Khludnev (@mkhludnev) (migrated from JIRA)

Dear Committers, Can't you move it forward? It's needed for our experiment with #6116 Although, there are a lot of things to do there.

asfimport commented 10 years ago

Paul Elschot (migrated from JIRA)

The recent addition of Accountable leads to some conflicts here, I'll post a new pull request later.

asfimport commented 10 years ago

ASF GitHub Bot (migrated from JIRA)

GitHub user PaulElschot opened a pull request:

https://github.com/apache/lucene-solr/pull/60

efbytesref as of 20140629

LUCENE-5524

This closes `#50`

Update to recent trunk.
Implement Accountable in EliasFanoDocIdSet.

There is one nocommit for the test of Accountable.ramBytesUsed on EliasFanoDocIdSet: disabled the test inherited from BaseDocIdSetTestCase because it fails even though there is a reasonable implementation.
The question is how to make this test pass: reimplement it to fit the implementation of Accountable in EliasFanoDocIdSet, or improve the implementation in EliasFanoDocIdSet. I'm probably overlooking something here.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/PaulElschot/lucene-solr efbytesref-201406a

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/60.patch

To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message:

This closes `#60`

commit 5ef98c1a655d9b79a01f8ffc884676076dbbec47 Author: Paul Elschot <paul.j.elschot@gmail.com> Date: 2014-06-29T09:40:04Z

efbytesref as of 20140629
asfimport commented 10 years ago

ASF GitHub Bot (migrated from JIRA)

GitHub user PaulElschot opened a pull request:

https://github.com/apache/lucene-solr/pull/62

efbytesref of 20140717, move Elias-Fano code to eliasfano package

efbytesref as of 20140717

LUCENE-5524

This closes `#60`

Update to recent trunk, and move all Elias-Fano classes into a separate package o.a.l.util.packed.eliasfano. The existing EliasFano\* classes are restored to trunk, the packed.eliasfano package can be merged back into  the packed package later.
Otherwise no changes to pull request 60.

The idea is to prepare a port to 4.9.0 and/or to prepare compilation into a separate jar.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/PaulElschot/lucene-solr efbytesref-201407a

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/62.patch

To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message:

This closes `#62`

commit 51a76c7789f38c2644f4019552a41c8199884557 Author: Paul Elschot <paul.j.elschot@gmail.com> Date: 2014-07-17T19:59:29Z

efbytesref of 20140717, move Elias-Fano code to eliasfano package
asfimport commented 10 years ago

ASF GitHub Bot (migrated from JIRA)

GitHub user PaulElschot opened a pull request:

https://github.com/apache/lucene-solr/pull/86

efbytesref of 20140817

LUCENE-5524

This closes `#62`

This is only an update to recent trunk

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/PaulElschot/lucene-solr efbytesref-201408a

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/86.patch

To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message:

This closes `#86`

commit 0af9647ae410688219367e1e445aa27abf485e4e Author: Paul Elschot <paul.j.elschot@gmail.com> Date: 2014-08-17T20:21:42Z

efbytesref of 20140817
asfimport commented 9 years ago

Paul Elschot (migrated from JIRA)

Update to trunk of today

asfimport commented 9 years ago

Adrien Grand (@jpountz) (migrated from JIRA)

Closing, we removed EliasFanoDocIdSet (see LUCENE-6484)

asfimport commented 9 years ago

ASF GitHub Bot (migrated from JIRA)

Github user PaulElschot closed the pull request at:

https://github.com/apache/lucene-solr/pull/86
asfimport commented 9 years ago

ASF GitHub Bot (migrated from JIRA)

Github user PaulElschot commented on the pull request:

https://github.com/apache/lucene-solr/pull/62#issuecomment-105225222

See LUCENE-5524 and #6689
asfimport commented 9 years ago

ASF GitHub Bot (migrated from JIRA)

Github user PaulElschot closed the pull request at:

https://github.com/apache/lucene-solr/pull/62
asfimport commented 9 years ago

ASF GitHub Bot (migrated from JIRA)

Github user PaulElschot closed the pull request at:

https://github.com/apache/lucene-solr/pull/60
asfimport commented 9 years ago

ASF GitHub Bot (migrated from JIRA)

Github user PaulElschot commented on the pull request:

https://github.com/apache/lucene-solr/pull/60#issuecomment-105225303

See LUCENE-5524 and #6689
asfimport commented 9 years ago

ASF GitHub Bot (migrated from JIRA)

Github user PaulElschot commented on the pull request:

https://github.com/apache/lucene-solr/pull/50#issuecomment-105225405

See LUCENE-5524 and #6689
asfimport commented 9 years ago

ASF GitHub Bot (migrated from JIRA)

Github user PaulElschot closed the pull request at:

https://github.com/apache/lucene-solr/pull/50
asfimport commented 9 years ago

ASF GitHub Bot (migrated from JIRA)

Github user PaulElschot commented on the pull request:

https://github.com/apache/lucene-solr/pull/45#issuecomment-105225478

See LUCENE-5524 and #6689
asfimport commented 9 years ago

ASF GitHub Bot (migrated from JIRA)

Github user PaulElschot closed the pull request at:

https://github.com/apache/lucene-solr/pull/45