kriszyp / lmdb-js

Simple, efficient, ultra-fast, scalable data store wrapper for LMDB
Other
484 stars 39 forks source link

getRange() in reverse mode skips the first record #80

Closed 1N50MN14 closed 2 years ago

1N50MN14 commented 2 years ago

This behaviour showed up for me after upgrading from 1.5.3 to 1.6.x.

Given the following range options that I have:

{
  start:'0Sdts8FwTqt2Hv5j9KE7ebjsQcFbYDdL/0Sdu5zqpvRiJBvOJG3CLCPGjULWtA5Ri',
  end:  '0Sdts8FwTqt2Hv5j9KE7ebjsQcFbYDdL/00000000MMsoFg0R8824kvws2YbpA0Vo',
  reverse: true
}

The above range query misses the first matching key (my keys consist of uuid / base62 timestamp encoded string - always correctly in order).

Running the same range query with reverse set to false:

{
  end:'0Sdts8FwTqt2Hv5j9KE7ebjsQcFbYDdL/0Sdu5zqpvRiJBvOJG3CLCPGjULWtA5Ri',
  start:  '0Sdts8FwTqt2Hv5j9KE7ebjsQcFbYDdL/00000000MMsoFg0R8824kvws2YbpA0Vo',
  reverse: false
}

Returns all matching keys without missing (the last) one.

Might be related to #69 - not sure, I'll follow up with a minimal code snippet later tonight.

kriszyp commented 2 years ago

Tried to reproduce this in the referenced commit (but it seems like it is working to me).

1N50MN14 commented 2 years ago

Ok so here's my store options:

{
  encoding: 'msgpack',
  useVersions: true,
  commitDelay: 1,
  compression: true,
  useStructuresKey: true
}

Put / range snippet:

//completely empty store
const store = env.openDB('dummy', opts);

store.putSync('0Sdts8FwTqt2Hv5j9KE7ebjsQcFbYDdL/0Sdtsud6g8YGhPwUK04fRVKhuTywhnx8', 
{
  id: '0Sdtsud6g8YGhPwUK04fRVKhuTywhnx8',
  aid: '0Sdts8FwTqt2Hv5j9KE7ebjsQcFbYDdL'
}, 1, null);

store.putSync('0Sdts8FwTqt2Hv5j9KE7ebjsQcFbYDdL/0Sdu0mnkm8lS38yIZa4Xte3Q3JUoD84V', 
{
  id: '0Sdtsud6g8YGhPwUK04fRVKhuTywhnx8',
  aid: '0Sdts8FwTqt2Hv5j9KE7ebjsQcFbYDdL'
}, 1, null);

const options =
{
  start: '0Sdts8FwTqt2Hv5j9KE7ebjsQcFbYDdL/0SdvKaMkMNPoydWV6HxZbFtKeQm5sqz3',
  end: '0Sdts8FwTqt2Hv5j9KE7ebjsQcFbYDdL/00000000dKZzSn03pte5dWbaYfrZl4hG'
  reverse: true
};

//yields one match missing key 0Sdts8FwTqt2Hv5j9KE7ebjsQcFbYDdL/0Sdtsud6g8YGhPwUK04fRVKhuTywhnx8
for (let { key, value } of store.getRange(options)) {} 

The above consistently replicates the behavior, whereas reverse false returns both keys as expected... going to test the above on 1.5.x to see how it behaves.

1N50MN14 commented 2 years ago

So I tested this on 1.5.3, 1.5.4, 1.5.5, 1.6.0-beta.1 and 1.6.0-beta.3 up to which it was working as expected - 1.6.0 is where getRange() starts to miss the first key.

kriszyp commented 2 years ago

Committed a fix; I am away from my computers for prebuilds for the next week, but could make a v1.6.4-pre release without prebuilds if you want (don't have prebuilds for linux anyway, which I think you are using?).

1N50MN14 commented 2 years ago

Oh thank you for the quick fix, yes please that would be very much appreciated (you're correct I'm on Linux)

kriszyp commented 2 years ago

ok, 1.6.4-pre should be published now. Let me know if you see any issue with it.

1N50MN14 commented 2 years ago

1.6.4-pre works like a charm! Thank you!