Nightly wheels and WASM/pyodide builds

[oscarbenjamin]

CC @agriyakhetarpal

See also: https://github.com/scientific-python/upload-nightly-action/issues/111

We should upload nightly wheels after each merge to main to the scientific-python nightly wheels index: https://anaconda.org/scientific-python-nightly-wheels

That is useful for downstream testing e.g. in SymPy. It would also make it possible to have the latest main branches of python-flint and SymPy available for testing in SymPy Live: https://live.sympy.org/

We should also build and test python-flint in WASM with pyodide in CI. There already are WASM builds of python-flint in pyodide: https://pyodide.org/en/stable/usage/packages-in-pyodide.html

Those can be used in the browser e.g. at: https://pyodide.org/en/stable/console.html

>>> import flint
>>> x = flint.fmpz_poly([0, 1])
>>> p = x**2 + 2*x + 1
>>> p
x^2 + 2*x + 1
>>> p.factor()
(1, [(x + 1, 2)])

It looks like cibuildwheel could build these with --platform pyodide: https://cibuildwheel.pypa.io/en/stable/options/

We would first need emscripten builds of GMP, MPFR and FLINT. The build recipes for those can be found in the pyodide repo e.g.: https://github.com/pyodide/pyodide/tree/main/packages/flint

Probably a separate CI workflow should be used for making and testing pyodide wheels because we can't upload them to PyPI and the build commands and CI setup would be a bit different. We can upload the WASM wheels to the nightly wheels index though and the final release versions are already built in pyodide.

[agriyakhetarpal]

Thanks for the mention, @oscarbenjamin! I'll be on holiday until early November, so I'm happy to step back and let someone else take it if you wish and if this is pressing for any reason. Otherwise, I can take a look at this in the coming weeks.

I'd suggest not using cibuildwheel for Pyodide builds, though – at least, not until I complete https://github.com/pypa/cibuildwheel/pull/2002. This is because cibuildwheel puts constraints around the Pyodide version available within it, and there is a chance it will be out of date until cibuildwheel releases a new version, so it's better to set up Pyodide and build the WASM wheels ourselves (for now). We don't have a long-term support policy for a particular Pyodide version, and have previously stated supporting just the latest available stable version. If having an older Pyodide version isn't a problem, then that's alright, the integration with cibuildwheel is quite neat.

[oscarbenjamin]

I'm happy to step back

I wasn't suggesting that you do the work here (although you are welcome to!).

Rather any advice is welcome:

so it's better to set up Pyodide and build the WASM wheels ourselves (for now).

Okay, good to know.

Do you know of any example of any similar project that has a CI workflow I could look at?

The task here is to build three dependent C libraries (GMP, MPFR and FLINT) that all build with autotools and then build the python-flint Python package on top. I'm not sure I understand how this works in pyodide but the other wheels all need to bundle those C dependencies in which is done by cibuildwheel's repair step (auditwheel, delvewheel, delocate).

I don't know if it matters but on current main python-flint uses meson-python which is different from the most recent release (which uses setuptools).

If having an older Pyodide version isn't a problem, then that's alright, the integration with cibuildwheel is quite neat.

The cibuildwheel integration would be more useful if we could treat the wheels like the others which we can't if we can't upload them to PyPI or if the build and setup steps are quite different. I guess in future using cibuildwheel will make more sense.

[agriyakhetarpal]

I wasn't suggesting that you do the work here (although you are welcome to!).

Thank you! I wasn't too sure about it because you pinged me here, so I'll help out wherever I can. :D

The task here is to build three dependent C libraries (GMP, MPFR and FLINT) that all build with autotools and then build the python-flint Python package on top. I'm not sure I understand how this works in pyodide but the other wheels all need to bundle those C dependencies in which is done by cibuildwheel's repair step (auditwheel, delvewheel, delicate).

Do you know of any example of any similar project that has a CI workflow I could look at?

I'd say that there is indeed a lack of documentation around navigating build systems with cross-compilation using Pyodide. Perhaps we need to publish more example packages across various build backends in a repository somewhere (this exists for just scikit-build-core + pybind11-based packages, but I'm unable to find the link to that repository – I'll update this comment with it when I can).

As for repairing the WASM wheel, we ship a tool called auditwheel-emscripten, which is integrated with the pyodide CLI using pyodide-build (you may try it with pip install pyodide-build && pyodide auditwheel --help). Right now, we repair the wheels with this interface immediately after they are built, for convenience. I'd also note that Emscripten does not fully support runtime search paths, yet: https://github.com/emscripten-core/emscripten/issues/22126, but that might not be relevant too much here.

I just wanted to clarify, though—for the C libraries you mentioned: GMP, MPFR and FLINT—will they be used during python-flint's build process as build-time/host dependencies, or are Python bindings as wheels for them used as runtime dependencies? If it's the former (I assume that's the more common scenario out of the two), then I'll have to think how to do it out-of-tree, since pyodide build outside the Pyodide repository works better for Python packages compared to shared and static libraries. It should still be doable, though, but the implementation might not be as neat. It might be easier if versions used for these C libraries remain relatively stable, since using their nightly builds for a nightly python-flint wheel would be quite disruptive (😅)

You could look at NumPy's CI workflow for a project that uses meson-python: https://github.com/numpy/numpy/blob/020e78cc3a4c7283bbe83290f0be5de3bc842f07/.github/workflows/emscripten.yml. Although, at the time of writing, it uses cibuildwheel and that abstracts away most of the build procedure, so you could take a look at one of my PRs from earlier this year where we re-enabled it: numpy/numpy#25894. I'd think adding support to python-flint's packaging infrastructure is something we can do without much hassle (or so I hope). Therefore, it shouldn't matter much that python-flint uses meson-python now (which I think as a decision is great!).

The cibuildwheel integration would be more useful if we could treat the wheels like the others which we can't if we can't upload them to PyPI or if the build and setup steps are quite different. I guess in future using cibuildwheel will make more sense.

I agree!

[oscarbenjamin]

I just wanted to clarify, though—for the C libraries you mentioned: GMP, MPFR and FLINT—will they be used during python-flint's build process as build-time/host dependencies, or are Python bindings as wheels for them used as runtime dependencies?

It is both. We need the headers at build time and then we need the shared libraries at runtime. There are no Python packages for GMP, MPFR and FLINT apart from python-flint itself which uses all of them (and needs to include all three bundled in to the wheel). The wheel build process is basically:

1. Build C lib GMP
2. Build C lib MPFR against GMP
3. Build C lib FLINT against GMP and MPFR
4. Build python-flint extension modules and wheel against GMP, MPFR and FLINT.
5. Repair the wheel to bundle GMP, MPFR and FLINT (and something like pthreads as well).

I will try this out locally when I get time with pyodide auditwheel to see how it works. Then I will see if I can get it working in CI (thanks for the NumPy links!).

[agriyakhetarpal]

I understand now, thanks for the context! When I get the time, I'll be glad to help out more, too, and I will try my best to answer any questions.

[oscarbenjamin]

Uploading nightly wheels was added in gh-235.

Now if we add a WASM build in CI we can add it to the upload job.