mtth / avsc

Avro for JavaScript :zap:
MIT License
1.27k stars 147 forks source link

Replace Buffer with Uint8Array #452

Open valadaptive opened 6 months ago

valadaptive commented 6 months ago

I still need to do some cleanups and replace uses of Buffer.compare, but I'm putting this PR here so you can benchmark it. Before I can complete this, #428 (minus the version bump) should be merged in, as the services code uses Buffer everywhere and expects fixed/bytes types to decode to Buffers.

I've made some tweaks to the benchmarking setup, so comparing this directly to the master branch won't work:

I've cherry-picked those benchmarking changes into the bench-tweaks branch, which you can use to compare benchmarks.

mtth commented 6 months ago

Thanks @valadaptive! I'll try to find time to merge #428.

mtth commented 5 months ago

FYI @valadaptive - #428 is in.

valadaptive commented 5 months ago

Working on removing Buffer usage from types.js now. I noticed the isJsonBuffer function, which seems to check if a given object is the JSON representation of a Buffer. Under what circumstances are Avro types directly serialized to JSON and/or parsed back directly? I can't easily polyfill Uint8Array to stringify to a regular array, so I'll probably need to insert a fixup step when stringifying/parsing.

valadaptive commented 1 day ago

@mtth Do you intend to support the ability to roundtrip various Avro types to/from JSON? With the current representation that uses Buffers, this works mostly fine, but with Uint8Array, the JSON representation is a lot more bloated:

> JSON.stringify(Buffer.from([1, 2, 3, 4, 5]))
'{"type":"Buffer","data":[1,2,3,4,5]}'
> JSON.stringify(new Uint8Array([1, 2, 3, 4, 5]))
'{"0":1,"1":2,"2":3,"3":4,"4":5}'

I lean towards removing the coerceBuffers option entirely to discourage people from serializing Uint8Arrays to JSON.