Guidance for packaging packages providing Python bindings

[bgamari]

Issue description

Packages like kicad and notmuch are currently rather awkward in that they are C/C++ programs, but provide Python bindings. Currently there is no documentation in the nixpkgs users guide offering guidance on how these packages should be packaged. Given that these bindings are currently quite hard to use in scripts (requiring that the user manually set PYTHONPATH appropriately), it seems this should change.

The documentation should answer this question: How does one package a package such that, for instance, nix run nixpkgs.notmuch -c python -c 'import notmuch' will actually work?

[veprbl]

I used to just add pythonPath attribute to my derivation to achieve PYTHONPATH setting. It seems like this is now out of favour: https://github.com/NixOS/nixpkgs/commit/985f0ce15dc1e19a200959450dff8add32bdeac9 I guess this new approach would require using pythonPackages.foo instead of foo.

[teto]

a bit of explanation was given in https://github.com/NixOS/nixpkgs/pull/40470 (basically add a py output) and then use toPythonModule from python-packages.nix

[veprbl]

So there is mkDerivation, buildPythonPackage and buildPythonApplication. There are also toPythonModule and toPythonApplication. I'm a bit confused about when to use these.

[FRidh]

Right, understandable:

• stdenv.mkDerivation is the standard builder. While you can use it for Python stuff, you'll need to do some things manually.
• buildPythonPackage is used in case the package is a library or used in a Python environment.
• buildPythonApplication is used in case the package is a stand-alone application, i.e., users have no intention of importing provided modules in a script.
• toPythonModule is used to convert a stdenv.mkDerivation derivation to buildPythonPackage. Primary use case is bindings of packages.
• toPythonApplication is to convert a buildPythonPackage derivation to buildPythonApplication. The primary use case is indeed a package is used as both library and application. In that case it will be added with buildPythonPackage in python-packages.nix and then with pythonPackages; toPythonApplication mypkg; in all-packages.nix.

[FRidh]

I've updated the docs in 39e9de1b85f1bec437333ed684cb0dc8cfe715c8. I could add this little overview as well if it is helpful.

[veprbl]

@FRidh If you don't want into too deep detail in the docs, maybe just copy your sentences into the source code near the definitions of the functions?

[FRidh]

Packages like kicad and notmuch are currently rather awkward in that they are C/C++ programs, but provide Python bindings.

• kicad: I couldn't find any docs explaining how the Python bindings are supposed to be build
• notmuch: these are available as python.pkgs.notmuch

[lopsided98]

Is toPythonModule supposed to work with nix-shell? When I try it, it doesn't even add Python to the PATH, let alone add the package to the PYTHONPATH. How does PYTHONPATH get set in nix-shell?

[FRidh]

@lopsided98

$ nix-shell -p "python3.withPackages(ps: with ps; [ rpm ])" --run "python3 -c 'import rpm; print(rpm)'"
<module 'rpm' from '/nix/store/jm2kfp1fcaz80sifhslysrzh0m22ms42-python3-3.6.6-env/lib/python3.6/site-packages/rpm/__init__.py'>

Python is added to PATH because it's propagated by most Python packages. We still need to get rid of that, so you can't rely on all packages having it anyway. PYTHONPATH is set by the Python interpreters' setup hook, and thus requires Python to be added through propagation or with -p.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/make-a-derivation-for-a-kicad-plugin/7317/3

[FRidh]

kicad bindings in bc3c3b4a59c7fc65e52ce1f617b24a30c0b03c05, note I have not tested them.

[evils]

@FRidh i suspect that won't work as kicad is a wrapper for kicad.base, which has /share/kicad/scripting/ with a bunch of python scripts kicad only exposes /bin with wrapped binaries, which pull in the libraries (3d models etc)

as these libraries add up to a few GB, i suspect using just kicad.base for python.pkgs.kicad is the better approach
or even passing through a separated kicad.py output as previously suggested here

i'm not familiar with python bindings though i'll try to get a test setup for plugins

[FRidh]

kicad.src is used, so not the wrapper.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[FRidh]

A common pattern these days is to put, when possible, the bindings in a separate output, .python. This works often fine, but not always. Specifically, it will not work when the bindings need to be found by pip (that is when the package is a build input in a buildPython*, and the package also has a .dev output. When a .dev is created, propagated-build-inputs file goes in there. We explicitly select .python and thus other outputs don't get added. We could potentially set propagatedBuildOutputs.

Probably a better method is to use multiple derivations. Either make an entire rebuild or, when possible, output a wheel in say a dist output, and then have a second derivation that uses buildPython* to install the wheel.

https://github.com/NixOS/nixpkgs/pull/193188#issuecomment-1265264255

[bluecmd]

kicad bindings in bc3c3b4, note I have not tested them.

I'm trying to use these but I am not able to do so. Seems PYTHONPATH is not updated to include the kicad module?

$ nix-shell -p python3Packages.kicad --run 'python3 -c "import sys, pprint; pprint.pprint(sys.path); import pcbnew"'
 '/nix/store/45dy86b5s375nqxksyc3w4lfawvdfs16-python3.11-cffi-1.15.1/lib/python3.11/site-packages',
 '/nix/store/fcb5xf03kxbfqrsh743ndzbcds940z1r-python3.11-pycparser-2.21/lib/python3.11/site-packages',
 '/nix/store/2pgjqb0m26i3k1lxlqnrmpf2a8if4yxi-python3-3.11.4/lib/python3.11/site-packages',
 '/nix/store/ksy60awn16la1mqbh89xijsc2fkc1y6c-python3.11-requests-2.29.0/lib/python3.11/site-packages',
 '/nix/store/66kwcfj9lpi1rfwwk028amknj4z2mdyd-python3.11-brotlicffi-1.0.9.2/lib/python3.11/site-packages',
 '/nix/store/jmjrxxpd4j8q5i8zx1gazmr7adw1rdhs-python3.11-certifi-2022.12.07/lib/python3.11/site-packages',
 '/nix/store/9bsjdiapg3asf97bih4v4qfyxfqvccx1-python3.11-charset-normalizer-3.0.1/lib/python3.11/site-packages',
 '/nix/store/a2497jzylmp8jy0jy2ddy5ymvn0gpgjp-python3.11-idna-3.4/lib/python3.11/site-packages',
 '/nix/store/ifpninh9rbpxvyixbqwv5gfx42c8d69a-python3.11-urllib3-1.26.14/lib/python3.11/site-packages',
 '/nix/store/h96ikah9achyvvh05v09nyfysji11zws-python3.11-brotli-1.0.9/lib/python3.11/site-packages',
 '/nix/store/mypm1ng0xi2nb79mj74fx807ml8qyvdb-python3.11-pysocks-1.7.1/lib/python3.11/site-packages',
 '/nix/store/04vqj99m9lpghg4jwg9lgv5gizkq91rz-python3.11-pyyaml-6.0/lib/python3.11/site-packages',
 '/nix/store/2pgjqb0m26i3k1lxlqnrmpf2a8if4yxi-python3-3.11.4/lib/python311.zip',
 '/nix/store/2pgjqb0m26i3k1lxlqnrmpf2a8if4yxi-python3-3.11.4/lib/python3.11',
 '/nix/store/2pgjqb0m26i3k1lxlqnrmpf2a8if4yxi-python3-3.11.4/lib/python3.11/lib-dynload']
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pcbnew'

[doronbehar]

I'm trying to use these but I am not able to do so. Seems PYTHONPATH is not updated to include the kicad module?

You are not using a proper command. Try this:

nix shell --impure --expr 'with builtins.getFlake "nixpkgs"; with legacyPackages.${builtins.currentSystem}; python3.withPackages(ps: [ps.kicad])' --command python