Closed vweevers closed 2 years ago
Updated the above with a more complete plan, that removes the need for serialization and asBuffer
options, and also paves the way for typed arrays.
Example of a transcoder (some should be hardcoded to optimize them, others like json+buffer
can be dynamic):
exports['utf8+view'] = new Encoding({
encode (data) {
return ArrayBuffer.isView(data) ? data : textEncoder.encode(data)
},
decode (data) {
return textDecoder.decode(data)
},
type: 'utf8+view',
format: 'view'
})
Doing this in abstract-leveldown
could have performance benefits too. For example, if we know that a particular value encoding on a get()
operation is idempotent (like buffer
or utf8
) and we know that the implementation supports that encoding "natively", then we can do _get(..., callback)
without wrapping the callback in another function that decodes the value.
Ugh, this works better than I expected. Now subleveldown
can just forward operations to a db, without unwrapping or rewrapping that db with encoding-down
or levelup
. Doesn't matter what encoding options that db used, or whether it stores data as buffers or Uint8Array internally. On the subleveldown
db you can use any encoding too, including Uint8Array even though subleveldown
internally works with Buffers and strings. It just fucking works.
The following tests are passing (locally; haven't pushed yet):
@ralphtheninja @juliangruber @MeirionHughes ARE YOU EXCITED? Because I am! Fuck!
Sounds like this gets us a lot of flexibility and api simplicity as well. Well done! 👏
Added support of other ecosystem encodings (codecs
, abstract-encoding
, multiformats
) to level-transcoder
, fixed some bugs, and updated its README: https://github.com/Level/transcoder. That part is now done.
As written in https://github.com/Level/community/issues/58, there are too many forms of encoding now. Yet for an
abstract-leveldown
implementation there's no builtin primitive to decode/deserialize data.So here's a WIP plan for refactoring. I'll be editing this.
format
property to each encodingutf8
,buffer
,view
,idjson
it will beutf8
, forutf8
it will beutf8
, etcbinary
encoding tobuffer
, withbuffer
as an alias (in limited places, i.e. where we already supported this)view
encoding that works with typed arrays (including Buffer because that's a UInt8Array)utf8
encoding to always return a string, because return types of.encode()
must be predictableabstract-leveldown
encoding.format
to the private APIput('a', 123, { valueEncoding: 'json' })
could result in_put('a', '123', { valueEncoding: 'utf8' })
level
, the default encoding will beutf8
abstract-leveldown
via the manifestmemdown
will set{ encodings: { buffer: true } }
which says "I want buffers"memdown({ storeEncoding: 'view' })
which will tellmemdown
to set{ encodings: { view: true } }
- andmemdown
won't have to do any translation itselfutf8
tobuffer
..type
will be e.g.utf8+buffer
,json+view
abstract-leveldown
.memdown
you'll seedb.supports.encodings.json
even thoughmemdown
itself doesn't handlejson
asBuffer
,valueAsBuffer
,keyAsBuffer
_serializeKey
,_serializeValue
abstract-leveldown
will lookup a transcoder.memdown
, adb.get(9)
will result in_get(<Buffer 39>)
by passing it trough the transcoderutf8+buffer
.In theory we can also support native encodings. If an implementation advertises e.g.encodings: { json: true }
, we can bypassabstract-leveldown
encoding. But if the encoding is idempotent (meaning.type === .format
) we won't, in order to still normalize user input.ascii
,ucs2
andutf16le
buffer
optional, becauseutf8
is supported by both Buffer and TextEncoder{ code: LEVEL_ENCODING_NOT_SUPPORTED }
, rather than falling back to theid
(identity) encoding