Python: wrapped python packages don't propagate their PYTHONPATH

[copumpkin]

Related to #23639 but not quite the same.

If a python package launches python, the child's PYTHONPATH will know nothing of the dependencies of the parent. This is almost never what you want. I think this is because rather than setting PYTHONPATH, we use site inside python to set the path for the current interpreter. Spawning new child interpreters won't get any of that stuff.

My current inclination would be to add a very simple line to the wrapper shim we have today:

os.environ['PYTHONPATH'] = ":".join(sys.path)

Right after we call reduce to add all the paths. That would ensure that all children inherit the right environment. Any thoughts?

cc @FRidh

[FRidh]

Originally we used --prefix PYTHONPATH in which case child processes do inherit PYTHONPATH but when one had several programs with different PYTHONPATH calling eachother it caused problems. Therefore, @abbradar proposed using --set PYTHONPATH which solved that issue, but I was against it because a) it breaks usage of a Python feature, PYTHONPATH, and b) certain applications couldn't be used anymore (like ipython). Furthermore, it broke the installCheckPhase.

Several days ago I tested another solution where I added a sitecustomize.py to the interpreter. That file is automatically loaded by the interpreter and in it I let it check for NIX_PYTHONPATH and then add that to site or sys.path. Then, we could use --set NIX_PYTHONPATH without it breaking usage of PYTHONPATH. Throughout Nixpkgs we should then use NIX_PYTHONPATH instead of PYTHONPATH.

---

Instead of NIX_PYTHONPATH we could also let it check for a .pth file in /nix/store/<package>/nix/python.pth where the pth file would contain the site-packages of all dependencies.

Also, about not breaking the usage of PYTHONPATH. We should take into account the distinction between applications and libraries. In the case of an application, say borg, it should work regardless of what we set PYTHONPATH or PYTHONHOME to in our current environment. But if we have a program that is used in a Python environment for development, like pytest, then I would expect it to take into account PYTHONPATH.

How do we deal with such issues in Nixpkgs in general?

[0xABAB]

@FRidh I like your partial analysis of the problem space. One way to make progress on this is to write down all the use cases (and not some subset, however appealing that is for people who have a problem that needs to be fixed "now") and only after there is agreement on that, generate candidate solutions for those, and then finally implement one of those.

One output of the use cases phase would be end-user documentation specifying what is possible. There should also be integration tests showing that these actually work (and linked from the documentation).

Doing it in the above way would make sure that at some point the Python integration is actually completed, as opposed to merely improved.

I am not against incremental temporary hacks which do not destroy the possibility of another proper design later, but it should not introduce distributed assumptions, which a good design wouldn't have.

[FRidh]

@0xABAB I agree that we should put this on paper. It has taken me quite a lot of time to get a good understanding of Python packaging in general, and how we deal with it on Nix. A while ago I wrote down a bit. It's quite brief and already partially outdated so I've now begun updating it. It's very much work in progress :)

[mmahut]

Any news on this issue?

[FRidh]

Subprocess should not inherit PYTHONPATH because it will cause breakage when other Python applications inherit PYTHONPATH. We now have NIX_PYTHONPATH in a sitecustomize.py that is automatically unset, thus preventing leaking. This is currently only used in buildEnv. We should use this eventually elsewhere as well.

Related tickets:

• 16591

• 11423 (which kind of requests the opposite)

[tbenst]

If I need subprocess to inherit PYTHONPATH, is there a way to accomplish this (besides using nix-shell)? See https://github.com/NixOS/nixpkgs/pull/74091. I'm happy to patch every call to subprocess, but seems like anything I pass for the environment is clobbered.

[stale[bot]]

Thank you for your contributions. This has been automatically marked as stale because it has had no activity for 180 days. If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity. Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on irc.freenode.net.

[ghost]

Important to me!

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/python-program-cant-invoke-module-in-same-package/54969/2