clearSync crashes node

[jbsiddall]

Hello!

i've a script that crashes node. filing this to either report bug or get schooling in how to use lmdb :P

I've a foo.ts script:

import { open } from 'lmdb'

const env = open('mydb', {})
console.log("pre clear")
env.clearSync()
console.log("post clear")

const n = 30_000

for (let i = 1; i < n; i++) {
  env.put(`${i}`, `${i}`)
}

and when run twice back to back rm -rf mydb && pnpm tsx ./foo.ts && pnpm tsx ./foo.ts causes node to crash on the second run with this error:

pre clear
post clear
pre clear
../dependencies/lmdb/libraries/liblmdb/mdb.c:2933: Assertion 'key.mv_size > 0' f
ailed in mdb_page_alloc()

first time the foo.ts is run, it creates db, and second time it runs it seems to crash on on env.clearSync().

I reran rm -rf mydb && pnpm tsx ./foo.ts && pnpm tsx ./foo.ts many times becuase i noticed unusual race conditions. one time it would output:

tsx ./foo.ts 
pre clear
post clear
pre clear

no errors but echo $? did output 139.

maybe 1 in 30 times of running it i get this error:

pre clear
post clear
pre clear
mdb_page_alloc error
put failed bad_sub, MDB_BAD_TXN: Transaction must abort, has a child, or is inva
lid: Invalid dupdata flag with no mc_xcursor
/home/joseph/Projects/playingcbor/node_modules/.pnpm/lmdb@3.2.0/node_modules/lmd
b/write.js:1008
                                                                env.commitTxn();
                                                                    ^

Error: MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
    at finishTxn (/home/joseph/Projects/playingcbor/node_modules/.pnpm/lmdb@3.2.
0/node_modules/lmdb/write.js:1008:13)
    at when (/home/joseph/Projects/playingcbor/node_modules/.pnpm/lmdb@3.2.0/nod
e_modules/lmdb/util/when.js:7:10)
    at LMDBStore.transactionSync (/home/joseph/Projects/playingcbor/node_modules
/.pnpm/lmdb@3.2.0/node_modules/lmdb/write.js:998:37)
    at LMDBStore.clearSync (/home/joseph/Projects/playingcbor/node_modules/.pnpm
/lmdb@3.2.0/node_modules/lmdb/open.js:344:9)
    at <anonymous> (/home/joseph/Projects/playingcbor/foo.ts:5:5)
    at Object.<anonymous> (/home/joseph/Projects/playingcbor/foo.ts:12:1)
    at Module._compile (node:internal/modules/cjs/loader:1369:14)
    at Object.transformer (/home/joseph/Projects/playingcbor/node_modules/.pnpm/
tsx@4.19.2/node_modules/tsx/dist/register-DCnOAxY2.cjs:2:1186)
    at Module.load (node:internal/modules/cjs/loader:1206:32)
    at Module._load (node:internal/modules/cjs/loader:1022:12) {
  code: -30782
}

Node.js v20.12.2

PS i know my project in the stacktrace is called playingcbor but cbor has no part to play in this script or bug.

attached project files for context:

code.zip

has 3 files: package.json, pnpm-lock.yaml, foo.ts, and mydb/ which has the db resulting from running rm -rf mydb && pnpm tsx ./foo.ts && pnpm tsx ./foo.ts.

I'm running this on nixos:

pnpm tsx --version
tsx v4.19.2
node v20.12.2

[jbsiddall]

found if i deleted node_modules and pnpm install and pnpm tsx ./foo.ts using node v22 this bug doesn't exist. Tried the same reinstall using node 20 and bug appears again.

[kriszyp]

So this potentially involves a node.js bug that has been resolved, you think?

[jbsiddall]

@kriszyp can you run this dockerfile and tell me if you get the same error? ideally you run it on a linux amd64 architecture:

from --platform=linux/amd64 node:22-slim
WORKDIR /app
run npm install -g pnpm
run pnpm install lmdb
RUN cat <<EOF > foo.mjs
import { open } from 'lmdb'

async function main() {
  const env = open('mydb', {})
  env.clearSync()

  for (let i = 0; i < 1_000_000; i++) {
    env.put(i, i)
  }
}

main()
EOF
cmd rm -rf ./mydb && node ./foo.mjs && node ./foo.mjs

docker build -t foo && docker run -it --rm foo. I still get error that i reported at the top of this. i also checked on a hetzner server and running this docker image on that also had same error.

[jbsiddall]

found that this dockerille works fine:

from --platform=linux/amd64 node:22-slim
WORKDIR /app
run npm install -g pnpm
run pnpm install lmdb
RUN cat <<EOF > foo.mjs
import { open } from 'lmdb'

async function main() {
  const old = open('mydb', {})
  const env = old.openDB('main')
  env.clearSync()

  for (let i = 0; i < 1_000_000; i++) {
    env.put(i, i)
  }
}

main()
EOF
cmd rm -rf ./mydb && node ./foo.mjs && node ./foo.mjs

Possible conclusion: bug with lmdb or library where 'clearing' the env fails, but clearing the database in the env works fine. Can i confirm that the lmdb env is meant to be clearable?

[jbsiddall]

okay wow i'm spamming this channel but epic new insight. on node v22, this bug only exists if the env's name isn't set. eg bug:

import { open } from 'lmdb'

async function main() {
  const env = open('mydb', {})
  env.clearSync()

  for (let i = 0; i < 1_000_000; i++) {
    env.put(i, i)
  }
}

no bug:

import { open } from 'lmdb'

async function main() {
  const env = open('mydb', {name: 'main'})
  env.clearSync()

  for (let i = 0; i < 1_000_000; i++) {
    env.put(i, i)
  }
}