Xonsh extensions are not importable in nested Xonsh session

charmoniumQ commented 6 months ago

Describe the bug

xonsh.override {
  extraPackages = ps: [ ps.psutil ];
};

Adding a package to the Xonsh environment using the above method seems to work initially, but it does not work for nested xonsh sessions (calling xonsh in xonsh).

This is especially salient for users who want to use xontribs and want to use xonsh as their nix shell (i.e., they typenix develop --command xonsh). The inner xonsh won't be able to use any of their xontribs, giving them a totally different experience than their native shell.

Steps To Reproduce

Steps to reproduce the behavior:

$ mkdir test
$ cd test
$ curl https://pastebin.com/raw/pz6a0ps5 > flake.nix
$ curl https://pastebin.com/raw/TA4R8kkr > flake.lock
$ nix build
$ export PATH=$PWD/result/bin:$PATH
$ xonsh -c 'import psutil'
$ xonsh -c 'xonsh -c "import psutil"'
xonsh: For full traceback set: $XONSH_SHOW_TRACEBACK = True
ModuleNotFoundError: No module named 'psutil'

Exporting path once, on the outside shell, is basically what home-manager does when you include xonsh in home.packages.

Expected behavior

Nested xonsh session should also be able to import psutil.

Additional context

result/bin/xonsh is a symlink to /nix/store/zdx...-python3-3.11.6-env/bin/xonsh which is a makeCWrapper around /nix/store/ri5...-xonsh-0.14.0/bin/xonsh, which is a makeWrapper around /nix/store/ri5...-xonsh-0.14.0/bin/.xonsh-wrapped.

The first, wrapper, /nix/store/zdx...-python3-3.11.6-env/bin/xonsh sets NIX_PYTHONPATH=/nix/store/zdx...-python3-3.11.6-env/lib/python3.11/site-package', in whichpsutil/*.py` lives.

The second wrapper, /nix/store/ri5...-xonsh-0.14.0/bin/xonsh prepends its containing directory to the PATH.

If the user addes result/bin to their PATH and types xonsh, /nix/store/zdx.../xonsh is invoked, psutil will be importable. But it will prepend to the path /nix/store/ri5...-xonsh-0.14.0/bin/xonsh to the PATH. In this shell, if they type xonsh again, /nix/store/ri5.../xonsh will be invoked and psutil will not be importable.

I am pretty sure the /nix/store/zdx.../xonsh wrapper should be the one to add itself to the PATH, which would fix this bug. It would also make invoking xonsh faster (fewer wrappers). OR none of the wrappers should add theirselves to the path, and it's the job of home-manager or the user to add xonsh to their PATH.

$ which xonsh
/home/sam/Downloads/test/result/bin/xonsh

$ readlink $(which xonsh)
/nix/store/zdx8436m5fma1xj6a8pjmlx1rbja35nx-python3-3.11.6-env/bin/xonsh

$ cat $(readlink $(which xonsh))
...
# ------------------------------------------------------------------------------------
# The C-code for this binary wrapper has been generated using the following command:

makeCWrapper '/nix/store/ri5v038ckppw0xcw3dgz8gv5ajaqpk0s-xonsh-0.14.0/bin/xonsh' \
    --set 'NIX_PYTHONPREFIX' '/nix/store/zdx8436m5fma1xj6a8pjmlx1rbja35nx-python3-3.11.6-env' \
    --set 'NIX_PYTHONEXECUTABLE' '/nix/store/zdx8436m5fma1xj6a8pjmlx1rbja35nx-python3-3.11.6-env/bin/python3.11' \
    --set 'NIX_PYTHONPATH' '/nix/store/zdx8436m5fma1xj6a8pjmlx1rbja35nx-python3-3.11.6-env/lib/python3.11/site-packages' \
    --set 'PYTHONNOUSERSITE' 'true'

# (Use `nix-shell -p makeBinaryWrapper` to get access to makeCWrapper in your shell)
# ------------------------------------------------------------------------------------
...

$ cat /nix/store/ri5v038ckppw0xcw3dgz8gv5ajaqpk0s-xonsh-0.14.0/bin/xonsh
#! /nix/store/q8qq40xg2grfh9ry1d9x4g7lq4ra7n81-bash-5.2-p21/bin/bash -e
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/kckqn8my6w4brmhscdh7l04706pbwz0x-python3.11-pygments-2.16.1/bin'':'/':'}
PATH='/nix/store/kckqn8my6w4brmhscdh7l04706pbwz0x-python3.11-pygments-2.16.1/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/ri5v038ckppw0xcw3dgz8gv5ajaqpk0s-xonsh-0.14.0/bin'':'/':'}
PATH='/nix/store/ri5v038ckppw0xcw3dgz8gv5ajaqpk0s-xonsh-0.14.0/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/5k91mg4qjylxbfvrv748smfh51ppjq0g-python3-3.11.6/bin'':'/':'}
PATH='/nix/store/5k91mg4qjylxbfvrv748smfh51ppjq0g-python3-3.11.6/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
export PYTHONNOUSERSITE='true'
exec -a "$0" "/nix/store/ri5v038ckppw0xcw3dgz8gv5ajaqpk0s-xonsh-0.14.0/bin/.xonsh-wrapped"  "$@" 

$ xonsh -c 'import sys; print("\n".join(sys.path))'
... stuff
/nix/store/zdx8436m5fma1xj6a8pjmlx1rbja35nx-python3-3.11.6-env/lib/python3.11/site-packages
... stuff

Yay, we have a sys.path that contains psutil

$ xonsh -c 'print("\n".join($PATH))'
... stuff
/nix/store/ri5v038ckppw0xcw3dgz8gv5ajaqpk0s-xonsh-0.14.0/bin
... stuff
/home/sam/Downloads/test/result/bin
... old $PATH here

Oh no! this puts the 'wrong' /nix/store/ri5... entrypoint above the one we just used from /home/sam/Downloads/result/bin.

$ xonsh -c 'which xonsh'
/nix/store/ri5v038ckppw0xcw3dgz8gv5ajaqpk0s-xonsh-0.14.0/bin/xonsh

That's the wrong entrypoint, because it does not set the NIX_PYTHONPATH to include /nix/store/zdx, which has psutil.

$ xonsh -c $'xonsh -c "import sys; print(\'\\\\n\'.join(sys.path))"'  
... /nix/store/zdx... path does not appear

Notify maintainers

@vrthra @adisbladis (because adisbladis wrote the xonsh wrapper).

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.64, NixOS, 24.05 (Uakari), 24.05.20231201.91050ea`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.19.2`
 - nixpkgs: `/home/sam/.nix-defexpr/channels/nixpkgs`

Add a :+1: reaction to issues you find important.

charmoniumQ commented 6 months ago

My workaround, which I don't like, is to add this to my rc.xsh:

# https://github.com/NixOS/nixpkgs/issues/276326
$PATH = [
    path for path in $PATH
    if not ((p"" / path / "xonsh").exists() and (p"" / path).parts[1] == "nix")
]

charmoniumQ commented 5 months ago

NixOS / nixpkgs