Unexpected end of buffer

vladar commented 3 years ago

I am seeing this error in one of my tests with lmdb-store:

  Error: In database datastore: Unexpected end of buffer

  - extractStrings

  - node.cjs:397 readString
    [test-site]/[msgpackr]/dist/node.cjs:397:22

  - node.cjs:197 read
    [test-site]/[msgpackr]/dist/node.cjs:197:10

  - node.cjs:480 readArray
    [test-site]/[msgpackr]/dist/node.cjs:480:14

  - node.cjs:335 read
    [test-site]/[msgpackr]/dist/node.cjs:335:12

  - node.cjs:736 recordDefinition
    [test-site]/[msgpackr]/dist/node.cjs:736:49

  - node.cjs:284 read
    [test-site]/[msgpackr]/dist/node.cjs:284:13

  - node.cjs:87 Packr.unpack
    [test-site]/[msgpackr]/dist/node.cjs:87:11

  - node.cjs:122 Packr.decode
    [test-site]/[msgpackr]/dist/node.cjs:122:15

  - index.js:335 Object.get
    [test-site]/[lmdb-store]/index.js:335:36

  - caching.js:37 Object.get
    [test-site]/[lmdb-store]/caching.js:37:17

  - lmdb-datastore.ts:88 getNode
    [test-site]/[gatsby]/src/datastore/lmdb/lmdb-datastore.ts:88:22

  - lmdb-datastore.ts:81
    [test-site]/[gatsby]/src/datastore/lmdb/lmdb-datastore.ts:81:20

  - ArrayLikeIterable.js:36 Object.next
    [test-site]/[lmdb-store]/util/ArrayLikeIterable.js:36:16

  - ArrayLikeIterable.js:27 Object.next
    [test-site]/[lmdb-store]/util/ArrayLikeIterable.js:27:34

  - Function.from

It happens when I run the following code:

const iterable = nodesDb
    .getKeys({ snapshot: false })
    .map(nodeId => getNode(nodeId))
    .filter(Boolean)

const result = Array.from(iterable);

So it lazily iterates all items in a database by key and fetches values by this key. But then at some point, it fails with this msgpackr error.

kriszyp commented 3 years ago

Do you have any idea what the (serialized) object is that might be triggering the error?

vladar commented 3 years ago

I am trying to reproduce it but it's tricky as data comes from an external source and I have to wait a while to retry. I will report back when the next build finishes (or fails)

vladar commented 3 years ago

I've narrowed it down to the serialized value in the attached tmp.txt file. It fails with this simple script:

const fs = require("fs")
const { unpack } = require("msgpackr")

const bin = fs.readFileSync(`./tmp.txt`)
const result = unpack(bin) // throws here

Although I am not sure if it is an issue with unpacking or actually packing.

kriszyp commented 3 years ago

Thank you for the excellent test case. This should now be fixed in msgpackr v1.3.1, but I'll get a proper lmdb-store release that depends on it out soon (have some other things I am wrapping up with it).

kriszyp commented 3 years ago

Ok, should be fixed in v1.5.0.

vladar commented 3 years ago

Yup, seems to be working fine. Thanks!

kriszyp / lmdb-js

Unexpected end of buffer #47