Dump Buffer for TypedArrays (for compactness and efficiency)

[SilentCicero]

Hi all,

Coming from the browser focused world, I see many of the Level libraries don't really use a ton of Buffer facility, and considering Buffer's can be easily fitted (and are just a view to) typed arrays, why not just dump the massive Buffer library altogether.

This would make Level a much leaner construction for browser based or p2p apps like mine, and may even make processing at a micro level faster.

Thoughts?

[vweevers]

Some nuance: buffer is 147 bytes minified and gzipped. Calling it "massive" isn't fair, especially considering the interoperability that it enables. That said, I understand the need to shave off bytes wherever possible.

Unfortunately, the level ecosystem does use buffer heavily. The dozens of modules probably only use a small part of buffer's API, that's true, but switching to typed arrays (or restricting ourselves to UInt8Array's API) is nevertheless a huge breaking change and a huge effort.

[vweevers]

I'd like to keep this open as a reminder. And to allow folks to chime in; I don't want to give the signal "it's never gonna happen".

[SilentCicero]

Ah okay!

Well it would be really nice to use these existing lower level APIs vs buffer.

Our apps need to work in countries where the internet connection is poor even with CDNs, so every byte over the wire counts for performance and experience.

I love the level APIs and philosophy (aside from a few little things) and would love to see a move to a lighter version.

On Wed, May 20, 2020 at 7:40 AM Vincent Weevers notifications@github.com wrote:

I'd like to keep this open as a reminder. And to allow folks to chime in; I don't want to give the signal "it's never gonna happen".

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/Level/community/issues/90#issuecomment-631421418, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACK2CXSVQ7OT6B6W3TSZSSDRSO6T3ANCNFSM4M5DWZAA .

[MeirionHughes]

I'm not even sure you'd save anything in the long run - Buffer gives extensive encoder support - that would have to be moved in level* if its not on the Buffer.

If its something you're passionate about, I'd recommend forking abstract-leveldown and try passing a Buffer prototype through to its constructor, which would be used in-place of require'ing the existing Buffer ponyfill. doing it this way is the least painful approach I can think of - but give the user the option to pass a custom cut-down prototype explicit for their needs and stack.

I'm more concerned with multiple Buffer ponyfills more than anything - I'm pretty sure current build (webpack) is bundling another Buffer polyfill. Also better support for Uint8Array would be nice: in my custom encoder I have to wrap a Uint8Array in a Buffer else its re-encoded, for some reason.

[vweevers]

I'm not even sure you'd save anything in the long run - Buffer gives extensive encoder support - that would have to be moved in level* if its not on the Buffer.

The answer for that is to not use level, which is a convenience module that is unlikely to drop buffer. Folks that don't want buffer should instead do something like:

const compose = require('level-compose')
const factory = compose(leveldown, levelup)
const db = factory('./db')

Which is to say, those folks won't get encodings, unless we write an alternative to encoding-down (see below). Down the line, once abstract-leveldown reaches feature parity with levelup, folks can use leveldown (or other) directly.

WIP list of tasks to make buffer optional (feel free to edit):

• [x] Replace Buffer.isBuffer with is-buffer (https://github.com/Level/abstract-leveldown/commit/8f792349bb940bac47152c7f02fae92c081afac7)
• [ ] Determine what to do with asBuffer / keyAsBuffer / valueAsBuffer
• [ ] Add db.supports.buffer (level-supports) and use that in abstract tests
• [ ] Write a new module that, like level, exports either leveldown or level-js depending on environment. To get the same convenience as level but without buffer.
• [ ] Write a new module similar to encoding-down but with bring-your-own codecs
• [ ] Determine what to do in memdown, or write an alternative that is backed by typed arrays
• [ ] Determine what to do in level-js, oh boy

[vweevers]

Covered by https://github.com/Level/community/issues/102 and https://github.com/Level/transcoder. It will be possible to use either Uint8Array, Buffer, or both, and to exclude the buffer shim from browser bundles if you choose to use Uint8Array.