search

r-wasm / rwasm

Build R packages for WebAssembly and create a CRAN-like repo for distribution.
https://r-wasm.github.io/rwasm/
Other
54 stars 4 forks source link

Use xz compression instead of gzip to further shrink tarballs? #2

Open HenrikBengtsson opened 12 months ago

HenrikBengtsson commented 12 months ago

install.packages() supports other types of compressions than gzip, including xz. By serving tar.xz instead of tar.gz/tgz package files, I think you could shave off ~10% of the file sizes.

For example, https://repo.r-wasm.org/bin/emscripten/contrib/4.3/digest_0.6.32.1.tgz is 131,292 bytes whereas the xz version is 116,132 bytes.

georgestagg commented 11 months ago

This is a nice idea in principle, but I think there is a cost to be paid in the time it takes to decompress the data inside the Wasm runtime. I am beginning to suspect R's decompression routines lead to a non-trivial amount of time spent during package installation in webR.

In the coming weeks I plan to experiment with delivering packages in the form of Emscripten file_packager images. IIUC these images are not compressed by default (though they can be), but in principle a modern web server and browser are able to negotiate compression over the wire in any case, using more efficient schemes such as Brotli.

I believe letting the browser handle over-the-wire compression natively, rather than decompression in Wasm, might improve package installation performance enough to bear the cost of larger package images on disk.