`pixi shell` freezes for `environment-modules` package

jjjermiah commented 1 month ago

Checks

[x] I have checked that this issue has not already been reported.
[x] I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

➜ pixi init
✔ Created /home/jermiah/bhklab/jermiah/projects/environment-module-issue/pixi.toml

➜ pixi add environment-modules
✔ Added environment-modules >=5.1.1,<6

➜ pixi shell-hook
export PATH="/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/bin:/home/jermiah/.local/share/zinit/polaris/bin:/home/jermiah/.config/.pixi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
export CONDA_PREFIX="/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default"
export PIXI_PROJECT_VERSION="0.1.0"
export PIXI_IN_SHELL="1"
export PIXI_EXE="/home/jermiah/.config/.pixi/bin/pixi"
export PIXI_PROJECT_ROOT="/home/jermiah/bhklab/jermiah/projects/environment-module-issue"
export PIXI_PROJECT_NAME="environment-module-issue"
export PIXI_PROJECT_MANIFEST="/home/jermiah/bhklab/jermiah/projects/environment-module-issue/pixi.toml"
export CONDA_DEFAULT_ENV="environment-module-issue"
export PIXI_ENVIRONMENT_NAME="default"
export PIXI_ENVIRONMENT_PLATFORMS="linux-64"
export PIXI_PROMPT="(environment-module-issue) "
. "/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/etc/conda/activate.d/environment-modules-activate.sh"

# shellcheck disable=all
pixi() {
    local first_arg="$1"
    local cmd="$PIXI_EXE $*"

    eval "$cmd"

    case "$first_arg" in
        add|a|remove|rm|install|i)
            eval "$($PIXI_EXE shell-hook --change-ps1 false)"
            rehash # Clear the command hash table in zsh
            ;;
    esac
}

verbose shell command

➜ pixi shell -vvv
 INFO pixi_manifest::discovery: Found manifest in directory: "/home/jermiah/bhklab/jermiah/projects/environment-module-issue", continuing further.
DEBUG pixi_config: Loading config from /etc/pixi/config.toml
DEBUG pixi_config: Loading config from /home/jermiah/.config/pixi/config.toml
DEBUG pixi_config: Loading config from /home/jermiah/.config/.pixi/config.toml
 INFO pixi_config: Loaded config from: /home/jermiah/.config/.pixi/config.toml
DEBUG pixi_config: Loading config from /home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/config.toml
DEBUG pixi_config: Failed to load local config: /home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/config.toml (error: no file was found at /home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/config.toml)
 INFO pixi::environment: verifying prefix location is unchanged, with prefix file: /home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/conda-meta/pixi_env_prefix
 INFO pixi::lock_file::update: the lock-file is up-to-date
 INFO pixi::lock_file::update: Environment 'default' is up-to-date with lock file hash
DEBUG pixi::activation: Running activation script for Default
DEBUG pixi::cli::shell: Pixi environment activation:
{"PIXI_PROJECT_VERSION": "0.1.0", "MODULES_CMD": "/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/libexec/modulecmd.tcl", "PIXI_PROMPT": "(environment-module-issue) ", "CONDA_DEFAULT_ENV": "environment-module-issue", " _mlredir": "1", "MODULESHOME": "/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default", "BASH_FUNC__module_raw%%": "() {  eval \"$(/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/bin/tclsh /home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/libexec/modulecmd.tcl bash \"$@\")\";", "PIXI_ENVIRONMENT_NAME": "default", " _mlstatus": "$?;", "PIXI_ENVIRONMENT_PLATFORMS": "linux-64", "MODULEPATH": "/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/modulefiles", "LOADEDMODULES": "", "PATH": "/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/bin:/home/jermiah/.local/share/zinit/polaris/bin:/home/jermiah/.config/.pixi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin", "BASH_FUNC_ml%%": "() {  module ml \"$@", "CONDA_PREFIX": "/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default", "PIXI_PROJECT_NAME": "environment-module-issue", "PIXI_PROJECT_MANIFEST": "/home/jermiah/bhklab/jermiah/projects/environment-module-issue/pixi.toml", "PIXI_IN_SHELL": "1", "PIXI_EXE": "/home/jermiah/.config/.pixi/bin/pixi", "BASH_FUNC_module%%": "() {  local _mlredir=0;", " if [ \"$MODULES_REDIRECT_OUTPUT\" ": " '1' ]; then", "PIXI_PROJECT_ROOT": "/home/jermiah/bhklab/jermiah/projects/environment-module-issue", "MANPATH": "/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/share/man:"}
DEBUG rattler_shell::shell: Guessing the current shell is zsh. Parent process name: zsh and args: []
 INFO pixi::cli::shell: Starting shell: Zsh(Zsh)
DEBUG pixi::cli::shell: Starting shell 'zsh -l -i' with source command: ' . "/tmp/pixi_env_rzl.sh"'

"/tmp/pixi_env_rzl.sh" is different than pixi shell-hook?

➜ cat "/tmp/pixi_env_rzl.sh"
export PIXI_PROJECT_VERSION="0.1.0"
export MODULES_CMD="/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/libexec/modulecmd.tcl"
export PIXI_PROMPT="(environment-module-issue) "
export CONDA_DEFAULT_ENV="environment-module-issue"
export  _mlredir="1"
export MODULESHOME="/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default"
export BASH_FUNC__module_raw%%="() {  eval "$(/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/bin/tclsh /home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/libexec/modulecmd.tcl bash "$@")";"
export PIXI_ENVIRONMENT_NAME="default"
export  _mlstatus="$?;"
export PIXI_ENVIRONMENT_PLATFORMS="linux-64"
export MODULEPATH="/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/modulefiles"
export LOADEDMODULES=""
export PATH="/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/bin:/home/jermiah/.local/share/zinit/polaris/bin:/home/jermiah/.config/.pixi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
export BASH_FUNC_ml%%="() {  module ml "$@"
export CONDA_PREFIX="/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default"
export PIXI_PROJECT_NAME="environment-module-issue"
export PIXI_PROJECT_MANIFEST="/home/jermiah/bhklab/jermiah/projects/environment-module-issue/pixi.toml"
export PIXI_IN_SHELL="1"
export PIXI_EXE="/home/jermiah/.config/.pixi/bin/pixi"
export BASH_FUNC_module%%="() {  local _mlredir=0;"
export  if [ "$MODULES_REDIRECT_OUTPUT" =" '1' ]; then"
export PIXI_PROJECT_ROOT="/home/jermiah/bhklab/jermiah/projects/environment-module-issue"
export MANPATH="/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/share/man:"
echo '=== DONE ==='

Issue description

I am trying to convert the snakemake project to use pixi but whereas the tests used to work using conda and a conda yaml file the environment-modules package (feedstock repo) is causing some issues:

the pixi shell is hanging
the module command is not found, guessing the failed ENV variable settings fail to redirect paths to the .pixi/envs/default/Modules respective areas.

Any help here would be great

Expected behavior

Ideally being able to do a pixi run which module be able to find the modules

jjjermiah commented 1 month ago

Update:

though unrelated to https://github.com/prefix-dev/pixi/issues/2851#issuecomment-2589424775, eval "$(pixi shell-hook)" does successfully activate shell

is it normal for pixi shell-hook and the tmp_... file to be this different (as shown in the outputs above)?

edit:

despite activating the shell, it still has errors:

➜ eval $(pixi shell-hook)
export: not valid in this context: .

jjjermiah commented 1 month ago

I think ive traced the main issue to this line in the pixi shell-hook

. "/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/etc/conda/activate.d/environment-modules-activate.sh"

I believe the . notation here is failing to be interpreted as source as opposed to export

this works:

eval "$(pixi shell-hook | sed -E 's|^(\. )(.*)$|source "\2"|')"

jjjermiah commented 1 month ago

looks like theres the same issue with the openmpi package


➜ pixi add openmpi                                 
✔ Added openmpi >=5.0.7,<6

projects/environment-module-issue                                                                                                  
➜ pixi shell-hook 
export PATH="/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/bin:/home/jermiah/.config/.pixi/bin:/home/jermiah/.vscode-server/cli/servers/Stable-6609ac3d66f4eade5cf376d1cb76f13985724bcb/server/bin/remote-cli:/home/jermiah/.local/share/zinit/polaris/bin:/home/jermiah/.config/.pixi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/jermiah/.vscode-server/extensions/ms-python.debugpy-2025.4.1-linux-x64/bundled/scripts/noConfigScripts:/home/jermiah/.vscode-server/data/User/globalStorage/github.copilot-chat/debugCommand"
export CONDA_PREFIX="/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default"
export PIXI_PROJECT_ROOT="/home/jermiah/bhklab/jermiah/projects/environment-module-issue"
export PIXI_PROJECT_VERSION="0.1.0"
export PIXI_IN_SHELL="1"
export PIXI_PROJECT_NAME="environment-module-issue"
export PIXI_PROJECT_MANIFEST="/home/jermiah/bhklab/jermiah/projects/environment-module-issue/pixi.toml"
export PIXI_EXE="/home/jermiah/.config/.pixi/bin/pixi"
export CONDA_DEFAULT_ENV="environment-module-issue"
export PIXI_ENVIRONMENT_NAME="default"
export PIXI_ENVIRONMENT_PLATFORMS="linux-64"
export PIXI_PROMPT="(environment-module-issue) "
. "/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/etc/conda/activate.d/libxml2_activate.sh"
. "/home/jermiah/bhklab/jermiah/projects/environment-module-issue/.pixi/envs/default/etc/conda/activate.d/openmpi_activate.sh"

# shellcheck disable=all
pixi() {
    local first_arg="$1"
    local cmd="$PIXI_EXE $*"

    eval "$cmd"

    case "$first_arg" in
        add|a|remove|rm|install|i)
            eval "$($PIXI_EXE shell-hook --change-ps1 false)"
            rehash # Clear the command hash table in zsh
            ;;
    esac
}

projects/environment-module-issue                                                                                                  
➜ eval $(pixi shell-hook)
export: not valid in this context: .

wolfv commented 1 month ago

Hmm this is pretty strange!

I wonder if part of the issue is perhaps this:

export BASH_FUNC_module%%="() {  local _mlredir=0;"
export  if [ "$MODULES_REDIRECT_OUTPUT" =" '1' ]; then"

That does look strange.

Can you show the contents of the activation script that fails?

jjjermiah commented 1 month ago

Here is the output of pixi shell -vvv:

❯ pixi shell -vvv
 INFO pixi_manifest::discovery: Found manifest in directory: "/home/gpudual/bhklab/jermiah/debug/test-env-modules", continuing further.
DEBUG pixi_config: Loading config from /etc/pixi/config.toml
DEBUG pixi_config: Loading config from /home/gpudual/.config/pixi/config.toml
DEBUG pixi_config: Loading config from /home/gpudual/.config/.pixi/config.toml
 INFO pixi_config: Loaded config from: /home/gpudual/.config/.pixi/config.toml
DEBUG pixi_config: Loading config from /home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/config.toml
DEBUG pixi_config: Failed to load local config: /home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/config.toml (error: no file was found at /home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/config.toml)
 INFO pixi::environment: verifying prefix location is unchanged, with prefix file: /home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/envs/conda-meta/pixi_env_prefix
 INFO pixi::lock_file::update: the lock-file is up-to-date
 INFO pixi::lock_file::update: Environment 'default' is up-to-date with lock file hash
DEBUG pixi::activation: Running activation script for Default
DEBUG pixi::cli::shell: Pixi environment activation:
{"BASH_FUNC_module%%": "() {  local _mlredir=0;", "PIXI_PROJECT_NAME": "test-env-modules", " _mlredir": "1", "MANPATH": "/home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/envs/default/share/man:", "PATH": "/home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/envs/default/bin:/Users/bhklab/.config/.pixi/envs/conda/bin:/home/gpudual/.local/share/zinit/polaris/bin:/home/gpudual/.config/.pixi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/gpudual/.modular/bin", "MODULES_CMD": "/home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/envs/default/libexec/modulecmd.tcl", " _mlstatus": "$?;", "CONDA_DEFAULT_ENV": "test-env-modules", "MODULEPATH": "/home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/envs/default/modulefiles", "PIXI_PROJECT_VERSION": "0.1.0", "PIXI_IN_SHELL": "1", "MODULESHOME": "/home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/envs/default", "PIXI_ENVIRONMENT_NAME": "default", "PIXI_EXE": "/home/gpudual/.config/.pixi/bin/pixi", " if [ \"$MODULES_REDIRECT_OUTPUT\" ": " '1' ]; then", "PIXI_PROJECT_MANIFEST": "/home/gpudual/bhklab/jermiah/debug/test-env-modules/pixi.toml", "PIXI_ENVIRONMENT_PLATFORMS": "linux-64", "PIXI_PROMPT": "(test-env-modules) ", "BASH_FUNC_ml%%": "() {  module ml \"$@", "BASH_FUNC__module_raw%%": "() {  eval \"$(/home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/envs/default/bin/tclsh /home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/envs/default/libexec/modulecmd.tcl bash \"$@\")\";", "PIXI_PROJECT_ROOT": "/home/gpudual/bhklab/jermiah/debug/test-env-modules", "CONDA_PREFIX": "/home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/envs/default", "LOADEDMODULES": ""}
DEBUG rattler_shell::shell: Guessing the current shell is zsh. Parent process name: zsh and args: []
 INFO pixi::cli::shell: Starting shell: Zsh(Zsh)
DEBUG pixi::cli::shell: Starting shell 'zsh -l -i' with source command: ' . "/tmp/pixi_env_wvW.sh"'

here is the output of the shell script referenced at the end of the pixi shell

❯ cat "/tmp/pixi_env_wvW.sh"
export BASH_FUNC_module%%="() {  local _mlredir=0;"
export PIXI_PROJECT_NAME="test-env-modules"
export  _mlredir="1"
export MANPATH="/home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/envs/default/share/man:"
export PATH="/home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/envs/default/bin:/Users/bhklab/.config/.pixi/envs/conda/bin:/home/gpudual/.local/share/zinit/polaris/bin:/home/gpudual/.config/.pixi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/gpudual/.modular/bin"
export MODULES_CMD="/home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/envs/default/libexec/modulecmd.tcl"
export  _mlstatus="$?;"
export CONDA_DEFAULT_ENV="test-env-modules"
export MODULEPATH="/home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/envs/default/modulefiles"
export PIXI_PROJECT_VERSION="0.1.0"
export PIXI_IN_SHELL="1"
export MODULESHOME="/home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/envs/default"
export PIXI_ENVIRONMENT_NAME="default"
export PIXI_EXE="/home/gpudual/.config/.pixi/bin/pixi"
export  if [ "$MODULES_REDIRECT_OUTPUT" =" '1' ]; then"
export PIXI_PROJECT_MANIFEST="/home/gpudual/bhklab/jermiah/debug/test-env-modules/pixi.toml"
export PIXI_ENVIRONMENT_PLATFORMS="linux-64"
export PIXI_PROMPT="(test-env-modules) "
export BASH_FUNC_ml%%="() {  module ml "$@"
export BASH_FUNC__module_raw%%="() {  eval "$(/home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/envs/default/bin/tclsh /home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/envs/default/libexec/modulecmd.tcl bash "$@")";"
export PIXI_PROJECT_ROOT="/home/gpudual/bhklab/jermiah/debug/test-env-modules"
export CONDA_PREFIX="/home/gpudual/bhklab/jermiah/debug/test-env-modules/.pixi/envs/default"
export LOADEDMODULES=""
echo '=== DONE ==='

note: I had to open a new session while the pixi shell was frozen to access the script

wolfv commented 1 month ago

@jjjermiah - the problem seems to be lines like export if [ "$MODULES_REDIRECT_OUTPUT" =" '1' ]; then"

we need to understand how that happens ...

jjjermiah commented 1 month ago

@wolfv if this works using conda, should I look into it in pixi or in the recipe?

wolfv commented 1 month ago

I think this would work fine with eval $(pixi shell-hook).

When running the activations script manually, I can see that we don't handle the bash function case well.

In bash, after running the activation script, and then running printenv I can see this:

MODULES_CMD=/Users/wolfv/Programs/trash/eee/.pixi/envs/default/libexec/modulecmd.tcl
TERM_PROGRAM=iTerm.app
BASH_FUNC_ml%%=() {  module ml "$@"
}
BASH_FUNC_module%%=() {  local _mlredir=1;
 if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then
 if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then
 _mlredir=0;
 else
 if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then
 _mlredir=1;
 fi;
 fi;
 fi;
 case " $@ " in
 *' --no-redirect '*)
 _mlredir=0
 ;;
 *' --redirect '*)
 _mlredir=1
 ;;
 esac;
 if [ $_mlredir -eq 0 ]; then
 _module_raw "$@";
 else
 _module_raw "$@" 2>&1;
 fi
}
BASH_FUNC__module_raw%%=() {  eval "$(/Users/wolfv/Programs/trash/eee/.pixi/envs/default/bin/tclsh /Users/wolfv/Programs/trash/eee/.pixi/envs/default/libexec/modulecmd.tcl bash "$@")";
 _mlstatus=$?;
 return $_mlstatus
}

This should either be parsed properly as "multiline" env var or we should ignore it. I need to educate myself on the parsing rules here.

wolfv commented 1 month ago

I think we need to adjust the parse_env in rattler_shell to make it work.

However, it might need bigger adjustments if we want to be able to export the bash functions ... it would be cool for users that use bash to also get them re-imported, but in a first step we could ignore them and ask the user to use eval $(pixi shell-hook) instead if they want all the shell integration.

prefix-dev / pixi