filecoin-project / oni

👹 (DEPRECATED; see README) Project Oni | Network Validation
https://docs.google.com/document/d/16jYL--EWYpJhxT9bakYq7ZBGLQ9SB940Wd1lTDOAbNE
7 stars 5 forks source link

refactor: switch to brotli+base64 encoding for binary data #234

Closed alanshaw closed 4 years ago

alanshaw commented 4 years ago

~32% smaller vectors:

$ ./tvx suite-messages > vectors.brotli.b64.ndjson
$ ls -lah vectors.*
-rw-r--r--  1 alan  staff   119K 14 Aug 16:07 vectors.gzip.brotli.ndjson
-rw-r--r--  1 alan  staff   174K 14 Aug 11:31 vectors.gzip.hex.ndjson

Note, only ~3% smaller than gzip+base64.

refs #183

alanshaw commented 4 years ago

N.B. There is no pure go brotli writer (I found a project with a reader but no writer yet) so here we're using google's cbrotli which has additional tooling and library requirements.

raulk commented 4 years ago

Thanks for the experiment! In the call today, we decided to go with b64+gzip and discard the Brotli idea, since it introduces build time complexity due to the lack of a Golang-native Brotli implementation, and the little gain we get compared to gzip.