benbovy
commented
1 year ago @brendan-ward thanks for opening this issue. Nanobind has been briefly discussed in various places but it is better to gather the discussion here. Also good to know that the transition from pybind11 to nanobind went smoothly in your case.
A few thoughts:
- Smaller binaries would be nice
- I think one of the main benefits of using nanobind here would be its low-level interface. It would allow accessing the Python Geography types from C++ in a much cleaner way than the hacks used here to get the type information. This would help for efficient Python <-> C++ conversion of Geography objects (#3).
- C++17 is fine I think, it is already used for other packages in the stack (s2geometry, s2geography, arrow, etc.) at least in conda-forge. Python 3.8+ is fine too for a new library like Spherely.
- I'm not sure either about nanobind support for numpy / dtypes, but one main limitation is that currently there's no equivalent of
pybind11::vectorize, which makes things super easy here for creating numpy ufunc-like functions. There's a few options:- Wait for nanobind to provide a
nb::vectorizehelper, although I don't know if that's part of their roadmap. - Copy the
pybind11::vectorizeimplementation in Spherely and adapt it to nanobind. This would probably help releasing the GIL in vectorized functions that create new Geography objects (#2). Thepybind11::vectorizeimplementation is quite complex, though, this is far from ideal if we have to maintain an adapted version here (even if simplified). - Use xtensor's
xt::vectorizealong with xtensor-python. The latter has no support for nanobind yet, though (https://github.com/xtensor-stack/xtensor-python/issues/284).
- Wait for nanobind to provide a
@benbovy feel free to close this if you are already aware of it and didn't want to consider it. I'm hoping to carve out a little time at some point to see what it might take to switch and whether or not it would be helpful; don't feel this issue is an obligation for you to do so.
nanobind is effectively a newer, smaller, lighter weight replacement of pybind11; the why page helps explain why it was created. The core API is very similar to pybind11 but at present is a bit less full-featured (which is partly a good thing). It targets a narrower range of C++ (17+) and Python (3.8+) versions. The potential downsides are that it is newer and less-full featured, and hasn't yet accumulated lots of GH issues / stack overflow posts around less common situations, compared to pybind11 (though these may not matter).
I'm not yet sure if the ndarray interface in nanobind would make some of the challenges around numpy and dtypes easier or harder.
I recently ported a small library from pybind11 to nanobind, and overall the experience was smooth and produced wheel files that were a little smaller (did not benchmark compile or runtime timings).