Pure JS `xz` compressing/decompressing

[sindresorhus]

xz is a lossless data compression program and file format which incorporates the LZMA/LZMA2 compression algorithms.

The current xz modules on npm uses native bindings, which is very fragile and not very suitable for reusable modules. It would be nice to have a pure JS version even though it would be a lot slower. Could then make a wrapper module that used the native binding if it installed successfully, otherwise the JS version.

Might be worth looking into compiling the native xz code to JS using emscripten.

This could potentially also be useful in the browser, where everything have to be pure JS.

Prior art

• Native binding:
  • https://www.npmjs.com/package/xz
  • https://www.npmjs.com/package/lzma-native
  • https://www.npmjs.com/package/node-liblzma
• JS
  • https://github.com/cscott/lzma-purejs (no idea if this is suitable for xz)

// @puzrin @mafintosh

[puzrin]

Personally, i'm more interested in high quality brotli port for ttf -> woff2 convertor.

[squarejaw]

My attempts to compile the xz code with emscripten have failed so far. At least some of the code uses multi-threading which is unsupported.

[sindresorhus]

@squarejaw You could compile with the --disable-threads flag, see the INSTALL file.

[squarejaw]

@sindresorhus That seems to fix part of it, but I'm still running into issues running the compiled js, e.g.

pipe() returning an error as we do not support them

If I remove some of the offending code calling pipe() it seems unable to find the files I specify to compress/decompress. I'll probably dig into it some more, but after looking at the spec and Java implementation it might not be too bad to write a JS version from scratch.

[sindresorhus]

@puzrin → https://github.com/devongovett/brotli.js

[sindresorhus]

it might not be too bad to write a JS version from scratch.

Sure, but it would most likely be slower and would require manual maintenance. Compiling it to JS with Emscripten would IMHO be the best option.

[puzrin]

@sindresorhus i know that project. It's cross-compiled, noticeable size - not interesting for me.

[puzrin]

Sure, but it would most likely be slower and would require manual maintenance. Compiling it to JS with Emscripten would IMHO be the best option.

That depends on your goals. If you need to get result fast - cross compiling is the best choice. But size will be 4x more than manually ported. Manually ported code is not slower, if you know how to do it. We did zlib porting as excercise to learn v8 optimizations.

[sindresorhus]

@puzrin I need this for Node.js so size isn't really that big of an issue. Performance is though. I know you could make it as fast, but most couldn't, and it would require a lot more work than with Emscripten.

[puzrin]

@sindresorhus you are right. In most cases, cross-compiling will be more optimal way to get predictable result in minimal time.

As i said, we had different and "crazy" (not business friendly) criterias to do zlib port - "interesting", "learning", "nobody will repeat" and so on :)

[simlu]

Any update on this?

[Richienb]

@sindresorhus Is https://www.npmjs.com/package/lzma what you're looking for?

[simlu]

@Richienb Unfortunately no. The lzma package also uses native bindings which exactly what the OP was trying to avoid

[pimterry]

https://www.npmjs.com/package/xzwasm was just published a couple of weeks ago, looks promising

[phanirithvij]

A side note: xzwasm only implements decompression