Open iFreilicht opened 2 months ago
The main issue is that building a dev environment with mkShell will either set the PYTHONPATH variable (if there are multiple pythons), or will leave it empty (if there is only one python).
This is a bit of a misunderstanding of what's happening here.
mkShell
isn't aware of whether an input is a Python interpreter or not.
What you're hitting is caused by how dependency propagation & setup hooks work.
This:
pythonEnv = pkgs.python3.withPackages (ps:
with ps; [
lxml
]);
returns a "closed" environment. No propagation of Python inputs happen from this derivation on.
However, when you add grc
, that is the bare return of buildPythonPackage
, a Python build with propagated dependencies, meaning you get the every propagated build inputs setup hook, and therefore a $PYTHONPATH
.
This dependency propagation is a long standing source of issues. I'm trying to fix the root cause in https://github.com/NixOS/nixpkgs/issues/272178.
Describe the bug
So here's a tricky one.
The main issue is that building a dev environment with
mkShell
will either set thePYTHONPATH
variable (if there are multiplepython
s), or will leave it empty (if there is only onepython
).This can lead to applications built with
buildPythonApplication
shadowing parts of PATH from the development environment, causing imports in python scripts to fail.Steps To Reproduce
The underlying bug that should be fixed can be reproduced with a simple flake:
nix develop -c make pythonpath
. It outputsPYTHONPATH=
grc
line fromflake.nix
nix develop -c make pythonpath
now outputsExpected behavior
I would expect the first invocation to output
Additional context
While it isn't strictly necessary to have a
PYTHONPATH
if only a single python derivation is an input tomkShell
, it can cause imports to fail when using python in certain environments.For example, you can try the following:
grc
anddirenv
installed viahome-manager
orconfiguration.nix
# grc
is commented out again)direnv allow
grc make run
. This will fail with an import errorgrc
line fromflake.nix
againgrc make run
again. Now, the script will succeed.The reason for this is that grc is also a python application, which of course doesn't know about the custom
python3.withPackages
in our development environment, and because there is noPYTHONPATH
, the different instance of python completely shadows all pythonPackages.This can be seen when running
grc make path
:The first path element is the python that
grc
depends on. So/usr/bin/env
inscript.py
will find that version, which isn't aware of thelxml
pythonPackage.This is resolved when
PYTHONPATH
is set, as evengrc
s python can now find thelxml
package.Fixing this might also improve the UX in other ways. For example, if you have jupyter installed gobally, you could then interact with the entire repository as expected without any additional work.
Notify maintainers
Are there any people with particular knowledge of python in nixpkgs? Those would be good candidates, I assume.
Metadata
Please run
nix-shell -p nix-info --run "nix-info -m"
and paste the result.Add a :+1: reaction to issues you find important.