JamesMGreene / nestdb

NestDB: An embedded persistent or in-memory database for Node.js, originally forked from NeDB
MIT License
58 stars 7 forks source link

Read file as Buffer to mitigate V8's 256MB max string size #6

Open JamesMGreene opened 7 years ago

JamesMGreene commented 7 years ago

Read file as Buffer, read lines as strings.

Implementation concept can be taken from existing NeDB PR: https://github.com/louischatriot/nedb/pull/493

However, it's not a terribly performant implementation since it is doing things like concatenating an array with itself rather than just pushing new items on to it, so a rewrite is in order.

This would mitigate frustrating issues like the following:

JamesMGreene commented 7 years ago

This looks like a better implementation to consider: https://github.com/louischatriot/nedb/pull/463

JamesMGreene commented 7 years ago

Seems like a symmetrical method for writing as a stream would also be necessary for the same reason.

JamesMGreene commented 7 years ago

The default writeAsStream implementation should be very mindful of honoring backpressure cues from Node like in the examples from the documentation and other sources, e.g. this [imperfect] StackOverflow thread.

JamesMGreene commented 7 years ago

IMPORTANT NOTE FOR BROWSER COMPATIBILITY

All Stream-specific module usage should be limited to "storage.js" (which is subbed out for the browser version, for example) and NOT put into "persistence.js" (which is NOT subbed out).

It is acceptable to have stream-related work in "persistence.js", e.g. listening for events, so long as it

JamesMGreene commented 6 years ago

For reference: https://github.com/nodejs/node/issues/3175