search

binast / binjs-ref

Reference implementation for the JavaScript Binary AST format
https://binast.github.io/binjs-ref/binjs/index.html
Other
433 stars 38 forks source link

Can we merge all dictionaries into single compressed stream? #397

Open arai-a opened 5 years ago

arai-a commented 5 years ago

Just an idea for the next format.

Currently prelude and content sections have multiple small streams (1 stream for each dictionary), and when decoding the file, decoder has to allocate buffer for each stream. also, currently there's no information about uncompressed size for brotli stream, afaik. That means decoder has to do allocation/reallocation multiple times while decoding prelude/content sections.

If there's no merit for compressing each stream, using single stream can make the decoder simpler. Also, adding uncompressed size might make it simpler (of course we need to validate the size before allocation tho)

Yoric commented 5 years ago

I agree, we definitely need to investigate this.

Yoric commented 5 years ago

I'll run a quick test on this.

Yoric commented 5 years ago

My quick test seems to indicate a small decrease in total file size. Unfortunately, I didn't write down a before/after, but it's encouraging.