Error: Attempt to read non-existing records

[futurGH]

Encountering a strange issue — there's a specific set of what I believe to be 5 consecutive records in my database that I'm calling "ghost records". Trying to read, write, or delete them is throwing the error Attempt to read non-existing records. Even setting their parent record to null causes an unending read lock on the record before eventually causing the same error.

I apologize for the vague issue without a repro, at the moment I'd just appreciate any advice on how to get rid of them! Trying to export the database, manually remove them, then import it back isn't working either; the export process seems to freeze upon reaching those records.

Attempt to read non-existing records
at NodeReader._treeDataReader (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/storage/binary/index.js:2933:19)
at BinaryReader.more (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/btree/binary-reader.js:110:38)
at BinaryReader.assert (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/btree/binary-reader.js:139:24)
at BinaryReader.get (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/btree/binary-reader.js:78:20)
at BinaryBPlusTree._readChild (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/btree/binary-tree.js:205:36)
at async leaf.getNext (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/btree/binary-tree.js:675:35)
at async createStreamFromBinaryTree (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/storage/binary/index.js:2629:56)
at async NodeReader.getAllocation (/project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/storage/binary/index.js:2179:9)
at async /project-name/node_modules/.pnpm/acebase@1.26.2/node_modules/acebase/dist/cjs/storage/binary/index.js:2187:45
at async Promise.all (index 0)

[futurGH]

Right after submitting this issue, I did manage to erase the records (and their sibling records as acceptable collateral damage) with

db.query("path/to/parent").forEach(snapshot => snapshot.ref.remove())

Leaving the issue up in case it ever becomes possible to consistently reproduce or if you have any idea what may be causing it.

[appy-one]

@futurGH thanks for reporting, I'll add more info to the error thrown to make an investigation easier. If you still have a backup with the "ghost records" in it, would be great if you can test with that code!

[futurGH]

Thanks for looking into this! I'm now getting:

Error: Attempt to read non-existing records of path "/maps/15000367/matches": 211 to 220 for index 27001 + 1024 bytes. Node has 211 allocated records in the following ranges: [object Object]

Couldn't get an object out of the [object Object], I'm pretty sure that's coming from the initial concatenation where the error is being thrown.

Let me know if you'd like me to send the db file.

[appy-one]

Thanks! Sending the db would be really appreciated! me@appy.one

[futurGH]

Email sent!

[appy-one]

@futurGH I managed to reproduce the issue with your db, digging into it now

[appy-one]

Ok, the issue seems to be that the B+Tree that stores all child properties has not been assigned sufficient amount of records to store its data, causing the leaf stored at the very end to have been cut off. I'm now investigating how that might have happened.

[appy-one]

@futurGH Is it possible your app ever crashed (eg because of an unhandled promise rejection under Node 15+), had a power failure while it was writing to your database, or because you manually terminated the process?

I'm onto a possible cause, where data for the B+Tree has to be moved elsewhere to allow growth. There would be an in-between state where the data hasn't moved yet, but the allocated size has already been assigned. If the app would have crashed in that specific moment, writing new data upon restart might have caused data to be written in unallocated space.

Let me know if that sounds like a possible cause

[futurGH]

A crash is entirely possible. The application is run with pm2 so I wouldn't have noticed it.

There was also a short period of time when writes were failing because the server was out of available storage space; now that I think about it, I only noticed these ghost records after that incident.

[appy-one]

Thanks for your feedback, a full drive could certainly also have caused this!

I'll focus on two things:

• Fix the issue by making it impossible to have (persistent) in-between state when moving data.
• Add repair option to db.recovery for this specific issue to recover as much data possible.

I'll probably be working on this on Monday, I'll keep you posted

[futurGH]

Much appreciated!

[appy-one]

@futurGH I've added a new recovery option to fix your db: await db.recovery.repairNodeTree('path/to/broken/collection')

Let me know if it works!

[futurGH]

Sorry for the delay, and thank you so much for putting this much work into my (probably self-induced) issue! I'm now getting:

Starting node tree repair for path "/maps/15000368"
Failed to repair node tree for path "/maps/15000368": Error: Node at maps/15000368 does not have a B+Tree key index

[appy-one]

@futurGH do it on the nodes you're unable to read specific children from. I've tested with the path you sent me through e-mail: "maps/15000367/matches"

[futurGH]

Whoops, my bad! It caught the broken records and successfully repaired them. No longer getting any errors trying to read — I can't know whether the data has been accurately restored but the structure is certainly correct. Thanks for the help!

[appy-one]

Awesome, closing the issue!