Compiling to a WebAssembly target

[agriyakhetarpal]

Hi there! I am trying to package, in specific, add a recipe for crc32c for Pyodide, a Python distribution compiled using the Emscripten toolchain to WebAssembly that enables running Python code in the browser. I thought that I should reach out here! Would a package maintainer or core developer for crc32c be able to help out on the compilation side of things? I imagine that with the help of the maintainers and author(s) here, this arduous task would be rendered much easier than with the efforts of a limited number of people who would otherwise be trying to figure this out alone.

I would be happy to share further context around this: the reason for this feature request is that version 3 of the Zarr Python package is going to use crc32c as a required dependency, and therefore wheels for crc32c for a wasm32 platform tag (with the ABI and ABI tags managed by the Emscripten version), which would be distributed through the Pyodide CDN across browsers and handled by Pyodide's infrastructure and developer tooling for building in-tree packages would be a wonderful addition towards bringing Zarr (and other dependents that might be using crc32c) to run on WASM-based platforms like JupyterLite.

---

xref: here is the corresponding issue on the Pyodide repository that I opened a while ago: https://github.com/pyodide/pyodide/issues/4787

[agriyakhetarpal]

I raised pyodide/pyodide#4789 and I think the recipe seems to build without issues with the Emscripten toolchain – I'm now waiting for the tests to run (and hopefully pass). Would one of the maintainers have sufficient bandwidth and like to be listed as a maintainer for the recipe as well, besides me?

[rtobar]

Many thanks @agriyakhetarpal for reaching out, and for the excellent overview of the context within which this sits.

I had a cursory look at the PRd and associated documents/files/etc, and whether with what you mentioned above it seems like the package is already compiling as expected? I saw some failures in some of the CI jobs in https://github.com/pyodide/pyodide/pull/4789, but I'm not familiar with the overall state of that pipeline; in a words, are those failures expected?

In terms of commitment, I can fully commit to enhance this package in any way it's required to compile in wasm platforms. OTOH I'd be hesitant to add myself as a maintainer of the pyodide recipe at this stage, since I'm 100% not familiar with their build process. I'm in a long period of leave ATM, but I might be able to play with it a bit, and maybe I'd gain some confidence. And again, changes to this package I'm more than happy to support.

Finally, as it turns out, we ourselves will probably end up using zarr in the near future in our of our projects, so it's nice this is coming back all the way.

[agriyakhetarpal]

Thank you for the warm response, @rtobar! Yes, the package is already compiling as expected and some of the CI jobs, quoting the words of the core Pyodide developers, are a bit flaky at some time or the other and therefore are known failures, even if they are expected ones or not. I think I just need to fix the crc32c-specific test cases there, which should be doable, and we should be good to go then.

In terms of commitment, I can fully commit to enhance this package in any way it's required to compile in wasm platforms. OTOH I'd be hesitant to add myself as a maintainer of the pyodide recipe at this stage, since I'm 100% not familiar with their build process. I'm in a long period of leave ATM, but I might be able to play with it a bit, and maybe I'd gain some confidence. And again, changes to this package I'm more than happy to support.

I understand your uncertainty about not wishing to be a maintainer for the Pyodide recipe, no worries about it – I can act as the de facto maintainer in that regard. The blog posts and the documentation are, IMO, excellent resources to read up on if you would wish to know more.

One way that crc32c would benefit would be if an out-of-tree Pyodide CI job could be added to this repository – this job would then install Pyodide, Emscripten, Node.js, and the necessary runtime dependencies, and then run the already existing test suite on PR builds or at a cadence you like, which would render Pyodide a semi-supported platform for this package for current versions and advances to later ones (as long as the CI job keeps passing).

I would be happy to set that up for you and maintain it for a few iterations of the bi-annual Pyodide releases as you see fit, please let me know! That said, it could also be overkill, though, since we are going to be using a subset of the test suite for the in-tree job, and given the size of the package, it might not be needed in essence. Indeed, Zarr is wonderful and to keep supporting it for WebAssembly would be great – getting crc32c in would mean we are halfway there (the rest of the required dependencies except one are already available)!

[agriyakhetarpal]

Successfully merged! crc32c should now be available as in-tree in the upcoming Pyodide 0.26.0 release, corresponding to Emscripten 3.1.58 and Python 3.12.1.

[rtobar]

Thanks again @agriyakhetarpal for spearheading this effort! I'll shy away from adding an off-tree job in this repository, specially now that the package is part of pyodide (great work!):

Welcome to the Pyodide 0.26.0 terminal emulator 🐍
Python 3.12.1 (main, May 27 2024 13:56:13) on WebAssembly/Emscripten
Type "help", "copyright", "credits" or "license" for more information.
>>> import crc32c
>>> 

With that, I'll close this issue, this is great news :tada: