Distribute PyGMT wheels with libgmt included

[weiji14]

Description of the desired feature

The core idea is to enable something like pip install pygmt-gmt6.4, where the GMT C binary is bundled directly with PyGMT. I.e. users don't have to install GMT first, and then install PyGMT, solving the issue of "Why can't I just pip install pygmt?".

distribute libgmt with PyPI wheels in the future (which would be awesome but difficult to achieve).

Originally posted by @leouieda in https://github.com/GenericMappingTools/pygmt/issues/1848#issuecomment-1080336677

I don't know how this fully works, but I know that there are Python packages like scipy and cupy which have C libraries bundled inside them. In the case of scipy, they've recently moved to this thing called Meson builds (see https://labs.quansight.org/blog/2021/07/moving-scipy-to-meson), and I'm wondering if there's something in there we can learn from

For cupy, if you look on https://pypi.org/project/cupy, there are several sub-packages like cupy-cuda114, cupy-cuda115, etc, whereby each cupy version is tied to a particular CUDA version binary. Again, not sure how this works but we might be able to have a similar setup where there's a plain pygmt source/wheel with no GMT, and a pygmt-gmt64, pygmt-gmt65, etc.

Are you willing to help implement and maintain this feature? Long term issue (>2023) that will need lots of teamwork.

[eli-schwartz]

In the case of scipy, they've recently moved to this thing called Meson builds (see https://labs.quansight.org/blog/2021/07/moving-scipy-to-meson), and I'm wondering if there's something in there we can learn from

SciPy has quite a lot of compiled code. So SciPy would have the advantage that it could just use Meson to statically build and link to its dependencies.

This project seems to use ctypes instead, hence the linked ticket mentions flit is a possible viable path forward. You can still use Meson, of course, but the biggest appeal for Meson is its excellent support for cross-language builds which isn't relevant here.

The only question seems to be how to get a shared library into the wheel so you can use ctypes with it. Meson can serve this role by including libgmt as an optional subproject and building PyPI wheels with that subproject which is instructed to install the library to python's site-packages, and the mesonpep517 build backend would bundle it in the wheel too.

But it seems like you could also just build a custom release script similar to auditwheel (which doesn't support detecting ctypes library dependencies) that just pops a copy of the library into the wheel after you build it using setuptools or flit.

You only need it for PyPI releasing anyway, right? Not necessarily for people building pygmt themselves.

[leouieda]

Thanks for the input @eli-schwartz! The problem is that libgmt has many dependencies on other shared libraries as well which are usually dynamically linked (at least the conda-forge build is). So we'd have to build a statically linked version of libgmt here to include it in the wheel.

[weiji14]

Twitter thread from @ocefpaf and @leouieda at https://twitter.com/leouieda/status/1570302518927785985 on potentially using the conda-forge feedstock (i.e. https://github.com/conda-forge/pygmt-feedstock) to build PyGMT wheels with libgmt. See examples at

• https://github.com/ocefpaf/netcdf4-win-wheels, wheels at https://github.com/ocefpaf/netcdf4-win-wheels/releases/tag/v1.6.1
• https://github.com/symengine/symengine-wheels, wheels at https://github.com/symengine/symengine-wheels/releases/tag/v0.9.2

Need to dig into the scripts to understand how all of this magic works.

[ocefpaf]

@weiji14 I'm not familiat with pygmt-gmt interaction. Does the former links and builds against the latter or it calls the CLI commands? If the it builds against then the approach above is the best one at the moment. If it calls the CLI one can keep pygmt pure python (noarch) and build a "fat" wheel for GMT using conda-press, making pygmt depend on that.

[weiji14]

Does the former links and builds against the latter or it calls the CLI commands? If the it builds against then the approach above is the best one at the moment. If it calls the CLI one can keep pygmt pure python (noarch) and build a "fat" wheel for GMT using conda-press, making pygmt depend on that.

PyGMT calls GMT C modules via ctypes, so I'm guessing the former? But the PyGMT and GMT builds are decoupled right now (i.e. one could build and install PyGMT without GMT, though it obviously won't run), so I'm not 100% sure which of these is correct.

The PyGMT conda-forge feedstock is currently noarch though. I'm not so sure I understand what a "fat" wheel for GMT would entail, are wheels for non-Python dependencies allowed on PyPI?

[ocefpaf]

PyGMT calls GMT C modules via ctypes, so I'm guessing the former?

Not exactly. You don't need GMT to build pygmt then, right? Only to run it, is that correct?

I'm not so sure I understand what a "fat" wheel for GMT would entail, are wheels for non-Python dependencies allowed on PyPI?

That is the right question! There are a few, like cmake, but they are all added as exceptions. I'm not sure how easy it would be to upload one for GMT. Maybe they don't care? The project to build wheels like that is kind of abandoned, see https://github.com/conda-incubator/conda-press, so I'm not sure it is the right path.

[weiji14]

PyGMT calls GMT C modules via ctypes, so I'm guessing the former?

Not exactly. You don't need GMT to build pygmt then, right? Only to run it, is that correct?

Yes, GMT isn't needed for PyGMT's build, they are only linked at runtime.

I'm not so sure I understand what a "fat" wheel for GMT would entail, are wheels for non-Python dependencies allowed on PyPI?

That is the right question! There are a few, like cmake, but they are all added as exceptions. I'm not sure how easy it would be to upload one for GMT. Maybe they don't care? The project to build wheels like that is kind of abandoned, see https://github.com/conda-incubator/conda-press, so I'm not sure it is the right path.

Looking at https://pypi.org/help, they don't really say what is or is not allowed on PyPI. You would think they only accept Python packages, but I'm not even sure how they would even enforce that :sweat_smile: Also not sure if this conda-press path is the one we want to take.

Maybe I'm missing something obvious, but what's the command to build a wheel in the netcdf4/symengine examples? I know python build_locally.py acts to create a .tar.bz2 file, but I don't know how to 1) output a .whl file and 2) have the C library be inside the .whl somehow.

[ocefpaf]

Maybe I'm missing something obvious, but what's the command to build a wheel in the netcdf4/symengine examples?

We need to document this approach better but here is the gist of it:

• up to this point it is the same process you are familiar with in conda-forge, except that we call pip wheel in line 11;
• here is the trick part b/c we need to "dealocate" all the DLLs used by the wheel. However, in https://github.com/ocefpaf/netcdf4-win-wheels/pull/4, I'm experimenting with delvewheel, which should make this part a bit simpler;
• we also ignore all the run exports from the dependencies

There is still a few details that we need to fix, like the minimum numpy in a recipe that uses numpy, it should be the one used in the build and not the one in the metadata. Delvewheel is also quite experimental and not officially recommended.

I would that, if conda-press was not abandoned, the approach of creating a wheel for GMT and another pure python one for pyGMT would be the best one.

[weiji14]

Cool, thanks for the pointers! I haven't got a Windows machine to try this out, but can probably figure my way around the shell script parts (once I find some spare time). Will also wait for some of the dust to settle first :laughing:

[weiji14]

Have been chatting with @kylebarron who's been trying out pixi (see e.g. https://github.com/kylebarron/geoarrow-rs/pull/177), which could be used for making cross-platform wheels that contain C libraries pulled from conda-forge:

curl -fsSL https://pixi.sh/install.sh | bash

cd pygmt/  # change into pygmt local repo
pixi init
pixi add gmt  # pulls from conda-forge
pip install pygmt  # or do `make install`

export GMT_LIBRARY_PATH=$(pwd)/.pixi/env/lib/
python -c "import pygmt; pygmt.show_versions()"

If we're building a 'fat' wheel, we could then copy libgmt.so/libgmt.dylib/gmt.dll (from conda-forge) into the wheel (for each platform) and distribute it, similar to what conda-press was doing. We could also try to create a wheel for libgmt, and keep the existing wheel for pygmt intact as recommended at https://github.com/GenericMappingTools/pygmt/issues/1853#issuecomment-1248535356, but would need to find a way to link both somehow.

[leouieda]

If we're building a 'fat' wheel, we could then copy libgmt.so/libgmt.dylib/gmt.dll (from conda-forge) into the wheel (for each platform) and distribute it, similar to what conda-press was doing.

Main thing to watch out for is that the wheel includes all of the dependency tree of libgmt as well. Otherwise it won't work. This is bound to be a big wheel.

We could also try to create a wheel for libgmt, and keep the existing wheel for pygmt intact as recommended at https://github.com/GenericMappingTools/pygmt/issues/1853#issuecomment-1248535356, but would need to find a way to link both somehow.

With this approach, I imagine that you would make a dummy PyPI package that has libgmt and then include it as a dependency of pygmt. But again, this would probably mean going down the dependency graph making wheels for each package.

[kylebarron]

Main thing to watch out for is that the wheel includes all of the dependency tree of libgmt as well.

I think that's usually pretty straightforward nowadays with delvewheel/auditwheel/delocate (windows/linux/mac). I haven't built a cffi wheel before, so it might be slightly different because the python code itself isn't linking against those libraries

[isuruf]

We could also try to create a wheel for libgmt, and keep the existing wheel for pygmt intact as recommended at https://github.com/GenericMappingTools/pygmt/issues/1853#issuecomment-1248535356, but would need to find a way to link both somehow.

No, that comment was about creating wheel for pygmt based on conda-forge pacakges.