PYTHONPATH is not available in environment when using python.buildEnv or python.withPackages

aswan89 commented 5 years ago

Issue description

When troubleshooting an issue (rstudio/reticulate#487 ) with the R package reticulate (designed to allow the use of python packages within R) it was discovered that the issue was related to reticulate relying on PYTHONPATH for finding the python packages of interest.

This was eventually solved with a call to mkShell and shellHooks to manually set PYTHONPATH but this seems like a counter-intuitive/hacky way to make python packages available to other software within a given environment.

I'm far from a competent python developer and I don't have a good understanding of what goes on to wrap python for use within Nix so it may be that there is an existing, less manual method for making python packages available to other software.

Steps to reproduce

Current documentation suggests a number of ways to make python package available for development purposes but given that this application required both R and Python a shell.nix file seemed to be the best option. Using this as a shell.nix file:

let pkgs = import <nixpkgs> {}; in
  let
  tarPyPacks = pkgs.python3.withPackages (p: [
    p.Keras p.numpy p.tensorflow
  ]);
  tarRPacks = pkgs.rWrapper.override { 
    packages = with pkgs.rPackages; [ 
      reticulate keras tensorflow 
    ];
  };
  in pkgs.mkShell {
    name = "forwardProgress";
    propagatedBuildInputs = [
      tarPyPacks
      tarRPacks
    ];
   }

echo $PYTHONPATH does not return a value, nor does os.environ['PYTHONPATH'] from within the python REPL.

Attempting to use reticulate functions within the generated environment causes a segfault, though I think that has more to do with their error handling.

Changing the shell.nix file as follows remedies the problem:

let pkgs = import <nixpkgs> {}; in
  let
  tarPyPacks = pkgs.python3.withPackages (p: [
    p.Keras p.numpy p.tensorflow
  ]);
  tarRPacks = pkgs.rWrapper.override { 
    packages = with pkgs.rPackages; [ 
      reticulate keras tensorflow 
    ];
  };
  in pkgs.mkShell {
    name = "forwardProgress";
    propagatedBuildInputs = [
      tarPyPacks
      tarRPacks
    ];
    shellHook = ''
       export PYTHONPATH="${tarPyPacks}/lib/python3.7:${tarPyPacks}/lib/python3.7/site-packages"
     '';
   }

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the results.

system: "x86_64-linux"
host os: Linux 4.19.36, NixOS, 19.03.172500.aade6ded796 (Koi)
multi-user?: yes
sandbox: no
version: nix-env (Nix) 2.2.2
channels(root): "nixos-19.03.172500.aade6ded796"
channels(andrew): "nixos-18.09beta11.c0bde692f6c"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

stale[bot] commented 4 years ago

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

Search for maintainers and people that previously touched the related code and @ mention them in a comment.
Ask on the NixOS Discourse.
Ask on the #nixos channel on irc.freenode.net.

alexvorobiev commented 3 years ago

This is important to my use case.

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

jbedo commented 1 year ago

For a shell I don't think you need the wrappers:

  mkShell {
    propagatedBuildInputs = [
      R
      rPackages.reticulate
      rPackages.keras
      python3
      python3Packages.keras
      python3Packages.tensorflow
    ];
  }

JordiSalaJuarez commented 1 year ago

If anyone is still looking at this, PYTHONPATH is not modified instead there is a separate NIX_PYTHONPATH environment variable containing all of the packages that need to be loaded with the environment, these packages are loaded by sitecustomize.py code, you can read more about it here

NixOS / nixpkgs