search

target / lorri

Your project's nix-env
Apache License 2.0
993 stars 69 forks source link

Difference between PYTHONPATH between lorri and nix-shell #71

Open pmiddend opened 5 years ago

pmiddend commented 5 years ago

I have a Python project with a shell.nix. When I...

I get a different path than when I...

Specifically, with nix-shell, PYTHONPATH begins with /run/user/1000/tmp.Jr4H7yM2Uz/lib/python3.7/site-packages. With lorri, it doesn't. It doesn't contain that path at all.

Profpatsch commented 5 years ago

You ran into https://github.com/target/lorri/issues/7 here

Shados commented 5 years ago

@Profpatsch this appears to be solved now that shellHooks are being run. However, when using a shell.nix that just returns a buildPythonPackage derivation with nix-shell, you also get any installed executables of that derivation available in PATH. This does not appear to be happening with lorri -- I am seeing a /run/user/$(id -u)/tmp.xxxxxxxxxx/lib/pythonX.Y/site-packages directory added to PYTHONPATH, I'm not seeing /run/user/$(id -u)/tmp.xxxxxxxxxx/bin added to PATH.

Profpatsch commented 5 years ago

I’m not so sure that we want to run shellHook at all, see https://github.com/target/lorri/issues/95

Profpatsch commented 4 years ago

(triage) Is this still a problem?

pmiddend commented 4 years ago

Yes, this is still an issue.

Profpatsch commented 4 years ago

Yes, this is still an issue.

Are you on a current (<2 months) version of lorri?

pmiddend commented 4 years ago

@Profpatsch Yes. Sorry for being brief earlier. I have now created a minimal test case for this:

https://github.com/pmiddend/lorri-python-env-test

You can check this repository out and test it as follows: 

ModuleNotFoundError: No module named 'mypackage'

This could, of course, have something do to with my specific system configuration. Or I’m not using shell.nix right, although the nixpkgs manual does state that this is the intended way to write it so you enter development mode for Python.

I hope this is at least reproducible and/or we can somehow debug this further.

curiousleo commented 4 years ago

Thank you very much for the repro case!

Here's what I get:

$ nix-shell
[...]
$ echo $PYTHONPATH | tr ':' "\n" | sort -u
/nix/store/crvfmdi6yhjim2f9y6zg7vs2lcn2gspv-python3.7-wheel-0.33.6/lib/python3.7/site-packages
/nix/store/hcdn4n9lkxpw3x5b05y5011zdfmrjvdz-python3.7-pip-19.2.3/lib/python3.7/site-packages
/nix/store/k5rdcbcwwpvj7l9f1yvd5mfggcfz16kk-python3-3.7.5/lib/python3.7/site-packages
/nix/store/r86byfrxrqfnm7s2dsiw0jwgy9nd668b-python3.7-setuptools-41.2.0/lib/python3.7/site-packages
/run/user/1000/tmp.GTQF6MAa0z/lib/python3.7/site-packages
$ exit
$ direnv allow
direnv: loading .envrc
Jan 16 08:52:21.967 INFO lorri has not completed an evaluation for this project yet, root: /home/leo/Code/lorri-python-env-test/shell.nix, lorri_version: 1
direnv: export +IN_NIX_SHELL
$ # wait a few seconds for lorri to build the environment, then hit <enter> so direnv can reload
direnv: loading .envrc
direnv: export +AR +AS +CC +CONFIG_SHELL +CXX +DETERMINISTIC_BUILD +IN_NIX_SHELL +LD +NIX_BINTOOLS +NIX_BINTOOLS_WRAPPER_x86_64_unknown_linux_gnu_TARGET_HOST +NIX_BUILD_CORES +NIX_BUILD_TOP +NIX_CC +NIX_CC_WRAPPER_x86_64_unknown_linux_gnu_TARGET_HOST +NIX_CFLAGS_COMPILE +NIX_ENFORCE_NO_NATIVE +NIX_HARDENING_ENABLE +NIX_INDENT_MAKE +NIX_LDFLAGS +NIX_LOG_FD +NIX_STORE +NM +OBJCOPY +OBJDUMP +PYTHONHASHSEED +PYTHONNOUSERSITE +PYTHONPATH +RANLIB +READELF +SIZE +SOURCE_DATE_EPOCH +STRINGS +STRIP +allowSubstitutes +buildInputs +builder +configureFlags +depsBuildBuild +depsBuildBuildPropagated +depsBuildTarget +depsBuildTargetPropagated +depsHostHost +depsHostHostPropagated +depsTargetTarget +depsTargetTargetPropagated +disallowedReferences +doCheck +doInstallCheck +extraClosure +name +nativeBuildInputs +origArgs +origBuilder +origExtraClosure +origOutputs +origPATH +origSystem +out +outputs +patches +postFixup +preHook +preferLocalBuild +propagatedBuildInputs +propagatedNativeBuildInputs +shell +stdenv +strictDeps +system ~LANG ~PATH
$ echo $PYTHONPATH | tr ':' "\n" | sort -u
/nix/store/crvfmdi6yhjim2f9y6zg7vs2lcn2gspv-python3.7-wheel-0.33.6/lib/python3.7/site-packages
/nix/store/hcdn4n9lkxpw3x5b05y5011zdfmrjvdz-python3.7-pip-19.2.3/lib/python3.7/site-packages
/nix/store/k5rdcbcwwpvj7l9f1yvd5mfggcfz16kk-python3-3.7.5/lib/python3.7/site-packages
/nix/store/r86byfrxrqfnm7s2dsiw0jwgy9nd668b-python3.7-setuptools-41.2.0/lib/python3.7/site-packages

The PYTHONPATH is the same modulo the temporary directory present when run with nix-shell.

pmiddend commented 4 years ago

@curiousleo Are you able to run the example in both cases?

curiousleo commented 4 years ago

I get the following:

With nix-shell

$ python mypackage/test.py
foo
$ python -m mypackage.test
foo

With lorri

$ python mypackage/test.py
Traceback (most recent call last):
  File "mypackage/test.py", line 1, in <module>
    from mypackage.othertest import x
ModuleNotFoundError: No module named 'mypackage'
$ python -m mypackage.test
foo
pmiddend commented 4 years ago

Okay, this is exactly the problem I'm seeing.

Profpatsch commented 4 years ago

This is set in a setupHook. I was of the opinion that we support those.

https://github.com/NixOS/nixpkgs/blob/d77b68e48709887741a3ff61c568beb9699aaac8/pkgs/development/interpreters/python/setup-hook.sh#L15-L17

cc @grahamc maybe we need to add another kind of setup hook primitive?

ryneeverett commented 4 years ago

@pmiddend's test case also demonstrates that the issue applies as much to PATH as to PYTHONPATH, as @Shados indicated. Lorri fails to add /run/user/$(id -u)/tmp.xxxxxxxxxx/bin to PATH as well.

Profpatsch commented 4 years ago

you can now use lorri shell to get exactly the same environment as in (an impure) nix shell: https://github.com/target/lorri/pull/168