Environment Variable replaced at Runtime

[ldoguin]

What happened?

I am setting up different variables to find binaries like

    "CARGO_HOME" : "/home/gitpod/.cargo",
    "GOPATH" : "/home/gitpod/go-packages",
    "GOROOT" : "/home/gitpod/go",
  },
  "remoteUser": "gitpod",
  "remoteEnv": {
    "PATH": "${containerEnv:PATH}:/home/gitpod/.cargo/bin:/home/gitpod/go/bin:/home/gitpod/go-packages/bin"

And unfortunately the /home/gitpod part is replaced by /workspace so all my binaries are not in the PATH anymore.

What did you expect to happen instead?

This works fine with codespace,

How can we reproduce the bug? (as minimally and precisely as possible)

You can reproduce by opening this repo in devpod and searching for go, tinygo, wit or other executable installed with Cargo: https://github.com/vados-cosmonic/wasmcon2024-couchbase-workshop/

[pascalbreuninger]

Hey @ldoguin, thanks for reporting the issue, we'll take a look!

[yacoob]

I'll drop in with my observations, as I think I'm seeing something similar. I've started debugging this more closely:

My test .devcontainer.json:

{
  "name": "debugging devpod's PATH problem",
  "image": "ghcr.io/yacoob/interactive:base",
  "containerEnv": {
    "PATH": "/containerEnv/marker:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "MARKER": "container"
  },
  "remoteEnv": {
    "PATH": "/remoteEnv/marker:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "MARKER": "remote"
  },
  "securityOpt": ["label=disable"],
  "remoteUser": "yacoob",
  "containerUser": "yacoob"
}

Test:

❯ podman exec reverent_kirch printenv PATH
/containerEnv/marker:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
❯ podman exec reverent_kirch printenv MARKER
container
❯ devpod ssh env-debug --command 'printenv PATH'
/home/yacoob/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin
❯ devpod ssh env-debug --command 'printenv MARKER'
remote
❯ devpod ssh env-debug --command 'cat /etc/envfile.json'
{"env":{"PATH":"/remoteEnv/marker:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","MARKER":"remote"}}

This strongly suggests that:

• containerEnv sets variables properly for the container (as podman exec shows both variables undisturbed)
• remoteEnv sets both variables via /etc/envfile.json, but PATH gets sanitized down to some standard value.

This is not totally unexpected, as traditional ssh also sanitizes PATH to its own default. Here, it's breaking things, as the values for containerEnv and remoteEnv can be set by either user or a feature they've included.

@pascalbreuninger any idea where that sanitization for PATH is happening?

[yacoob]

Further evidence:

❯ devpod ssh --set-env MARKER=/bla --set-env PATH=/workspaces/foo.venv/bin:/usr/bin --command 'printenv PATH' .
/home/yacoob/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin
❯ devpod ssh --set-env MARKER=/bla --set-env PATH=/workspaces/foo.venv/bin:/usr/bin --command 'printenv MARKER' .
/bla

Looks like devpod ssh - or the agent on the container's end - actually sanitizes PATH.

[yacoob]

One more thing: a (crude) workaround for me was to symlink the binaries I was missing into ~/.local/bin. As I'm making a Python devcontainer, I ended up having this:

"postCreateCommand": "(curl -LsSf https://astral.sh/uv/install.sh | sh) && uv sync && echo 'source ${containerWorkspaceFilder}/.venv/bin/activate' >> ~/.zshrc.local && ln -sf ${containerWorkspaceFolde
r}/.venv/bin/python3 ~/.local/bin/python3",

Not ideal, but does the trick in the meantime.

Correct fix (IMO) is to change devpod to honour PATH set via containerEnv and remoteEnv - as this seems to be the way for devcontainer features to hook into the container.