Transition `Pygrackle` from `setuptools` to `scikit-build-core`

[mabruzzo]

Description

This PR transitions Pygrackle's build-system from setuptools to scikit-build-core.

[!TIP] To quickly and easily install pygrackle with the new build-system, you can execute the following command:

$ pip install pygrackle@git+https://github.com/mabruzzo/grackle.git@scikit-build-core 

(This will only work if you have hdf5 installed in a standard way. It will overwrite any existing Pygrackle install and will compile a fresh-copy of Grackle that lives inside the Pygrackle module).

Motivation

As discussed during the June user/dev zoom call, moving to the scikit-build-core backend build-system from setuptools appears to be the correct path to getting a robust build-system that will allow us to put pygrackle on PyPI. The current setuptools backend is

Description

This PR transitions the build-backend from setuptools to scikit-build-core.

• scikit-build-core uses the configuration language specified by PEP 621 -- this logic is encoded at the pyproject.toml file at the root-level of the repository.[^1]
• This build-backend makes use of CMake instead of a python-script (this CMake logic can be found
• In other words, we can directly use the cmake build system introduced in #182 to build the grackle library as a part of Pygrackle
• This leads to a relatively robust installation process.

I have added documentation about the installation process. By default, an invocation of pip will compile a fresh-copy of Grackle (as a shared library) and package it inside Pygrackle. For testing purposes, the system also supports:

• linking against a copy of a Grackle shared library that was previously built with the classic build-system (all of the old idiosyncratic linking behavior was maintained here).
• linking against a copy Grackle shared library that was previously built with the CMake build-system. This works slightly differently from how setuptools linked to CMake-builds -- we now use idiomatic CMake functionality to link against the build-directory or against an installation.[^2] All of the nuances are described in the documentation.

EDIT: now that #182 is merged, this is ready for review!

[^1]: Originally the file was located at src/python/pyproject.toml, but I realized that was going to make distributing Pygrackle on PyPI in SDist-format more challenging. This prompted me to ask the scikit-build-core developers what to do and they said that the standard thing to do was to put pyproject.toml at the root of the directory. They also pointed out that this was necessary to let pip install directly from a GitHub repository (just like the command at the top of this PR).

[^2]: Because we no longer use the old setuptools-behavior I was able to remove a step in the build-system where we made a symlink of libgrackle-dev*.so to libgrackle.so. This was an unidiomatic choice and I'm happy to be rid of it before people come to depend upon this behavior (NOTE: we still symlink libgrackle-dev*.so to libgrackle.so in installation directories)

[^3]: There are some extra details that can be worked out related to keeping the build-directory around between builds to speed up rebuilds. Interestingly the h5py docs have a discussion about very similar details (even though it uses setuptools)

[ngoldbaum]

It's been a long time since I worked on grackle but I'm still subscribed to the repo and noticed the mention of not liking cmake much. Very much agreed there! Have you looked at all at meson, with meson-python as a build backend? IMO meson is much nicer than cmake and the meson scripting language is much easier to learn.

[mabruzzo]

Thanks for reaching out and your suggestion! And yes, while I personally don't mind cmake, I worry a little about mixing it with python build-system (since there is a bit of a learning curve). Recently, I did look at Meson -- it looks really nice! I totally agree that the meson's scripting language is really nice! If we were developing grackle in a vacuum, I would be inclined to give meson a shot.

In practice, there are currently a couple reasons to prefer building the core grackle library with cmake:

1. Most importantly, it makes grackle easier to consume. Downstream projects built with cmake have the option to perform (crude) automatic dependency management -- a downstream project can compile grackle as part of their build

   • In fact, if grackle is built with cmake, I think that downstream projects built with meson will also be able to do this same sort of thing
   • if grackle were built with meson, then downstream cmake projects would lose this ability.

2. Outside of python packaging (where meson has become popular), cmake is currently more widely used in general. For simulation codes in particular, I can name 3 that use cmake off the top of my head. I don't know of any that use meson.

And if we're currently building Grackle's core with cmake, I suspect it makes more sense to stick with it for Pygrackle compared to mixing meson-python with cmake (but I could be wrong).

[ngoldbaum]

Outside of python packaging (where meson has become popular), cmake is currently more widely used in general. For simulation codes in particular, I can name 3 that use cmake off the top of my head. I don't know of any that use meson.

That makes a ton of sense. Integrating well with cmake that is already used quite a lot in the ecosystem makes a lot of sense and scikit-build-core is definitely the way to integrate with python packaging if you need a cmake build.