Some `activate` tests fail on NixOS

[zmitchell]

On my NixOS machine it appears that my shell config is leaking into tests and causing tests to fail.

Furthermore, the suite hangs at

 activate runs hook and profile scripts only once

output from two failed tests

``` ✗ bash: interactive activate puts package in path [20225] tags: activate activate activate:path activate:path:bash (in test file activate.bats, line 223) `FLOX_SHELL="bash" USER="$REAL_USER" NO_COLOR=1 run -0 expect "$TESTS_DIR/activate/interactive-hello.exp" "$PROJECT_DIR"' failed, expected exit code 0, got 1 spawn /home/zmitchell/src/flox/watchdog-watch-more/cli/target/debug/flox activate --dir /tmp/nix-shell.p4h61C/bats-run-OQkdTw/test/5/project-5 parent: waiting for sync byte parent: telling child to go ahead parent: now unsynchronized from child spawn: returns {834847} expect: does "" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no ⠁ Preparing environment 'project-5'... expect: does "\u2801 Preparing environment 'project-5'... " (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no ⠁ Preparing environment 'project-5'... expect: does "\u2801 Preparing environment 'project-5'... \r\u001b[2K\u2801 Preparing environment 'project-5'... " (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? yes expect: set expect_out(0,string) "\u2801 Preparing environment 'project-5'... \r" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "\u2801 Preparing environment 'project-5'... \r" expect: continuing expect expect: does "\u001b[2K\u2801 Preparing environment 'project-5'... " (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no expect: does "\u001b[2K\u2801 Preparing environment 'project-5'... \r\u001b[2K" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? yes expect: set expect_out(0,string) "\u001b[2K\u2801 Preparing environment 'project-5'... \r" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "\u001b[2K\u2801 Preparing environment 'project-5'... \r" expect: continuing expect expect: does "\u001b[2K" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no ✅ You are now using the environment 'project-5'. To stop using this environment, type 'exit' expect: does "\u001b[2K\u2705 You are now using the environment 'project-5'.\r\nTo stop using this environment, type 'exit'\r\n\r\n" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? yes expect: set expect_out(0,string) "\u001b[2K\u2705 You are now using the environment 'project-5'.\r\nTo stop using this environment, type 'exit'\r\n\r\n" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "\u001b[2K\u2705 You are now using the environment 'project-5'.\r\nTo stop using this environment, type 'exit'\r\n\r\n" expect: continuing expect expect: does "" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no Sourcing .bashrc Setting PATH from .bashrc expect: does "Sourcing .bashrc\r\nSetting PATH from .bashrc\r\n" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? yes expect: set expect_out(0,string) "Sourcing .bashrc\r\nSetting PATH from .bashrc\r\n" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "Sourcing .bashrc\r\nSetting PATH from .bashrc\r\n" expect: continuing expect expect: does "" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no zmitchell in 🌐 chungus in test/5/project-5 via ❄️ impure (flox-dev-env) ❯ expect: does "\u001b[?2004h\r\n\u001b[1;33mzmitchell\u001b[0m in \u001b[1;2;32m\ud83c\udf10 chungus\u001b[0m in \u001b[1;36mtest/5/project-5\u001b[0m via \u001b[1;34m\u2744\ufe0f impure (flox-dev-env)\u001b[0m \r\n\u001b[1;32m\u276f\u001b[0m " (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? yes expect: set expect_out(0,string) "\u001b[?2004h\r\n\u001b[1;33mzmitchell\u001b[0m in \u001b[1;2;32m\ud83c\udf10 chungus\u001b[0m in \u001b[1;36mtest/5/project-5\u001b[0m via \u001b[1;34m\u2744\ufe0f impure (flox-dev-env)\u001b[0m \r\n" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "\u001b[?2004h\r\n\u001b[1;33mzmitchell\u001b[0m in \u001b[1;2;32m\ud83c\udf10 chungus\u001b[0m in \u001b[1;36mtest/5/project-5\u001b[0m via \u001b[1;34m\u2744\ufe0f impure (flox-dev-env)\u001b[0m \r\n" expect: continuing expect expect: does "\u001b[1;32m\u276f\u001b[0m " (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no expect: timed out Last output: spawn /home/zmitchell/src/flox/watchdog-watch-more/cli/target/debug/flox activate --dir /tmp/nix-shell.p4h61C/bats-run-OQkdTw/test/5/project-5 parent: waiting for sync byte parent: telling child to go ahead parent: now unsynchronized from child spawn: returns {834847} expect: does "" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no ⠁ Preparing environment 'project-5'... expect: does "\u2801 Preparing environment 'project-5'... " (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no ⠁ Preparing environment 'project-5'... expect: does "\u2801 Preparing environment 'project-5'... \r\u001b[2K\u2801 Preparing environment 'project-5'... " (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? yes expect: set expect_out(0,string) "\u2801 Preparing environment 'project-5'... \r" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "\u2801 Preparing environment 'project-5'... \r" expect: continuing expect expect: does "\u001b[2K\u2801 Preparing environment 'project-5'... " (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no expect: does "\u001b[2K\u2801 Preparing environment 'project-5'... \r\u001b[2K" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? yes expect: set expect_out(0,string) "\u001b[2K\u2801 Preparing environment 'project-5'... \r" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "\u001b[2K\u2801 Preparing environment 'project-5'... \r" expect: continuing expect expect: does "\u001b[2K" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no ✅ You are now using the environment 'project-5'. To stop using this environment, type 'exit' expect: does "\u001b[2K\u2705 You are now using the environment 'project-5'.\r\nTo stop using this environment, type 'exit'\r\n\r\n" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? yes expect: set expect_out(0,string) "\u001b[2K\u2705 You are now using the environment 'project-5'.\r\nTo stop using this environment, type 'exit'\r\n\r\n" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "\u001b[2K\u2705 You are now using the environment 'project-5'.\r\nTo stop using this environment, type 'exit'\r\n\r\n" expect: continuing expect expect: does "" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no Sourcing .bashrc Setting PATH from .bashrc expect: does "Sourcing .bashrc\r\nSetting PATH from .bashrc\r\n" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? yes expect: set expect_out(0,string) "Sourcing .bashrc\r\nSetting PATH from .bashrc\r\n" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "Sourcing .bashrc\r\nSetting PATH from .bashrc\r\n" expect: continuing expect expect: does "" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no zmitchell in 🌐 chungus in test/5/project-5 via ❄️ impure (flox-dev-env) ❯ expect: does "\u001b[?2004h\r\n\u001b[1;33mzmitchell\u001b[0m in \u001b[1;2;32m\ud83c\udf10 chungus\u001b[0m in \u001b[1;36mtest/5/project-5\u001b[0m via \u001b[1;34m\u2744\ufe0f impure (flox-dev-env)\u001b[0m \r\n\u001b[1;32m\u276f\u001b[0m " (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? yes expect: set expect_out(0,string) "\u001b[?2004h\r\n\u001b[1;33mzmitchell\u001b[0m in \u001b[1;2;32m\ud83c\udf10 chungus\u001b[0m in \u001b[1;36mtest/5/project-5\u001b[0m via \u001b[1;34m\u2744\ufe0f impure (flox-dev-env)\u001b[0m \r\n" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "\u001b[?2004h\r\n\u001b[1;33mzmitchell\u001b[0m in \u001b[1;2;32m\ud83c\udf10 chungus\u001b[0m in \u001b[1;36mtest/5/project-5\u001b[0m via \u001b[1;34m\u2744\ufe0f impure (flox-dev-env)\u001b[0m \r\n" expect: continuing expect expect: does "\u001b[1;32m\u276f\u001b[0m " (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no expect: timed out ✗ catalog: bash: interactive activate puts package in path [20246] tags: activate activate activate:path activate:path:bash (in test file activate.bats, line 235) `FLOX_SHELL="bash" USER="$REAL_USER" NO_COLOR=1 run -0 expect "$TESTS_DIR/activate/interactive-hello.exp" "$PROJECT_DIR"' failed, expected exit code 0, got 1 ✨ Created environment 'project-6' (x86_64-linux) Next: $ flox search <- Search for a package $ flox install <- Install a package into an environment $ flox activate <- Enter the environment $ flox edit <- Add environment variables and shell hooks ✅ 'hello' installed to environment 'project-6' spawn /home/zmitchell/src/flox/watchdog-watch-more/cli/target/debug/flox activate --dir /tmp/nix-shell.p4h61C/bats-run-OQkdTw/test/6/project-6 parent: waiting for sync byte parent: telling child to go ahead parent: now unsynchronized from child spawn: returns {835313} expect: does "" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no ⠁ Preparing environment 'project-6'... expect: does "\u2801 Preparing environment 'project-6'... " (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no expect: does "\u2801 Preparing environment 'project-6'... \r\u001b[2K" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? yes expect: set expect_out(0,string) "\u2801 Preparing environment 'project-6'... \r" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "\u2801 Preparing environment 'project-6'... \r" expect: continuing expect expect: does "\u001b[2K" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no ✅ You are now using the environment 'project-6'. To stop using this environment, type 'exit' expect: does "\u001b[2K\u2705 You are now using the environment 'project-6'.\r\nTo stop using this environment, type 'exit'\r\n\r\n" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? yes expect: set expect_out(0,string) "\u001b[2K\u2705 You are now using the environment 'project-6'.\r\nTo stop using this environment, type 'exit'\r\n\r\n" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "\u001b[2K\u2705 You are now using the environment 'project-6'.\r\nTo stop using this environment, type 'exit'\r\n\r\n" expect: continuing expect expect: does "" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no Sourcing .bashrc Setting PATH from .bashrc expect: does "Sourcing .bashrc\r\nSetting PATH from .bashrc\r\n" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? yes expect: set expect_out(0,string) "Sourcing .bashrc\r\nSetting PATH from .bashrc\r\n" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "Sourcing .bashrc\r\nSetting PATH from .bashrc\r\n" expect: continuing expect expect: does "" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no zmitchell in 🌐 chungus in test/6/project-6 via ❄️ impure (flox-dev-env) ❯ expect: does "\u001b[?2004h\r\n\u001b[1;33mzmitchell\u001b[0m in \u001b[1;2;32m\ud83c\udf10 chungus\u001b[0m in \u001b[1;36mtest/6/project-6\u001b[0m via \u001b[1;34m\u2744\ufe0f impure (flox-dev-env)\u001b[0m \r\n\u001b[1;32m\u276f\u001b[0m " (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? yes expect: set expect_out(0,string) "\u001b[?2004h\r\n\u001b[1;33mzmitchell\u001b[0m in \u001b[1;2;32m\ud83c\udf10 chungus\u001b[0m in \u001b[1;36mtest/6/project-6\u001b[0m via \u001b[1;34m\u2744\ufe0f impure (flox-dev-env)\u001b[0m \r\n" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "\u001b[?2004h\r\n\u001b[1;33mzmitchell\u001b[0m in \u001b[1;2;32m\ud83c\udf10 chungus\u001b[0m in \u001b[1;36mtest/6/project-6\u001b[0m via \u001b[1;34m\u2744\ufe0f impure (flox-dev-env)\u001b[0m \r\n" expect: continuing expect expect: does "\u001b[1;32m\u276f\u001b[0m " (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no expect: timed out Last output: spawn /home/zmitchell/src/flox/watchdog-watch-more/cli/target/debug/flox activate --dir /tmp/nix-shell.p4h61C/bats-run-OQkdTw/test/6/project-6 parent: waiting for sync byte parent: telling child to go ahead parent: now unsynchronized from child spawn: returns {835313} expect: does "" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no ⠁ Preparing environment 'project-6'... expect: does "\u2801 Preparing environment 'project-6'... " (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no expect: does "\u2801 Preparing environment 'project-6'... \r\u001b[2K" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? yes expect: set expect_out(0,string) "\u2801 Preparing environment 'project-6'... \r" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "\u2801 Preparing environment 'project-6'... \r" expect: continuing expect expect: does "\u001b[2K" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no ✅ You are now using the environment 'project-6'. To stop using this environment, type 'exit' expect: does "\u001b[2K\u2705 You are now using the environment 'project-6'.\r\nTo stop using this environment, type 'exit'\r\n\r\n" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? yes expect: set expect_out(0,string) "\u001b[2K\u2705 You are now using the environment 'project-6'.\r\nTo stop using this environment, type 'exit'\r\n\r\n" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "\u001b[2K\u2705 You are now using the environment 'project-6'.\r\nTo stop using this environment, type 'exit'\r\n\r\n" expect: continuing expect expect: does "" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no Sourcing .bashrc Setting PATH from .bashrc expect: does "Sourcing .bashrc\r\nSetting PATH from .bashrc\r\n" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? yes expect: set expect_out(0,string) "Sourcing .bashrc\r\nSetting PATH from .bashrc\r\n" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "Sourcing .bashrc\r\nSetting PATH from .bashrc\r\n" expect: continuing expect expect: does "" (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no zmitchell in 🌐 chungus in test/6/project-6 via ❄️ impure (flox-dev-env) ❯ expect: does "\u001b[?2004h\r\n\u001b[1;33mzmitchell\u001b[0m in \u001b[1;2;32m\ud83c\udf10 chungus\u001b[0m in \u001b[1;36mtest/6/project-6\u001b[0m via \u001b[1;34m\u2744\ufe0f impure (flox-dev-env)\u001b[0m \r\n\u001b[1;32m\u276f\u001b[0m " (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? yes expect: set expect_out(0,string) "\u001b[?2004h\r\n\u001b[1;33mzmitchell\u001b[0m in \u001b[1;2;32m\ud83c\udf10 chungus\u001b[0m in \u001b[1;36mtest/6/project-6\u001b[0m via \u001b[1;34m\u2744\ufe0f impure (flox-dev-env)\u001b[0m \r\n" expect: set expect_out(spawn_id) "exp6" expect: set expect_out(buffer) "\u001b[?2004h\r\n\u001b[1;33mzmitchell\u001b[0m in \u001b[1;2;32m\ud83c\udf10 chungus\u001b[0m in \u001b[1;36mtest/6/project-6\u001b[0m via \u001b[1;34m\u2744\ufe0f impure (flox-dev-env)\u001b[0m \r\n" expect: continuing expect expect: does "\u001b[1;32m\u276f\u001b[0m " (spawn_id exp6) match glob pattern "flox \[project-*\]"? no "*\n"? no "*\r"? no expect: timed out ```

[zmitchell]

I can't find the previous discussion on this topic, but in short, I think the issue was that my shell config leaks into the tests, so my prompt is not in the format that the tests are expecting and thus expect fails to get the desired output.

I do remember some discussion over whether we wanted the shell configs used in these tests to be completely hermetic, but I don't remember the outcome, nor can I find the discussion.

[zmitchell]

I discovered that the bash provided by the devshell is likely not the Bash that we intend tests to be run with:

✦ ❯ nix why-depends . /nix/store/717iy55ncqs0wmhdkwc5fg2vci5wbmq8-bash-5.2p32/bin/bash
path '/home/zmitchell/src/flox/rexpect/cli/flox-test-utils' does not contain a 'flake.nix', searching up
warning: Git tree '/home/zmitchell/src/flox/rexpect' is dirty
/nix/store/lng4rw59xkx1ps14l17gacgj0yc0d6n2-flox-1.3.4-dirty
└───/nix/store/lk8dcy5a6mls2br9hmjmmc5dyy56w152-flox-pkgdb-1.3.4
    └───/nix/store/717iy55ncqs0wmhdkwc5fg2vci5wbmq8-bash-5.2p32

If you add a bashInteractive dependency in shells/default.nix you get the correct shell. However, if I flox activate from within either of these bash shells I get my normal (e.g. not minimal) prompt (use imagination and pretend this is in color after flox activate):

bash-5.2$ flox activate
flox [rexpect]
zmitchell in 🌐 chungus in src/flox/rexpect on  zmitchell/rexpect-spike [!+] via ❄️  impure (flox-dev-env)
❯ # this breaks all kinds of test suite stuff

[zmitchell]

I discovered the root cause last week. If you have a shell configured via nix-darwin or home-manager, that config is placed in /etc/bashrc, which is loaded before ~/.bashrc. In the test suite we set $HOME to point at temporary directory to circumvent the user's .bashrc. This doesn't help in the Nix-configured case because their shell will still load /etc/bashrc.

One solution is to create a collection of wrappers for our test shells that set the appropriate --norc flags for each shell to ensure that a user's rc files aren't loaded at all. This has other side-effects though, since some tests are intentionally verifying something to do with rc files.