envdir not working with pyenv + virtualenvs properly? daemontools behaves correctly

[AlJohri]

hi @jezdez, it seems envdir is modifying the PATH in some unexpected way when using virtualenv. daemontools seems to be working as expected.

I have a pretty vanilla set up using pyenv and virtualenvwrapper. Nothing fancy.

setup:

$ deactivate
$ mkvirtualenv test
$ deactivate

with jezdez/envdir:

$ workon test
$ which python
/Users/johria/.virtualenvs/test/bin/python
$ envdir envs/prod which python
/Users/johria/.pyenv/versions/3.6.1/bin/python

^^^^ error is here ^^^^

uninstall jezdez/envdir, install daemontools:

$ deactivate
$ pip uninstall envdir
$ brew install daemontools

with daemontools:

$ workon test
$ which python
/Users/johria/.virtualenvs/test/bin/python
$ /usr/local/bin/envdir envs/prod which python
/Users/johria/.virtualenvs/test/bin/python

[blueyed]

Thanks for the repost.

NOTE: you are using pyenv apparently, and maybe even pyenv-virtualenvwrapper. So two additional failure points.

[blueyed]

If you're using pyenv, why does which python not point at ~/.pyenv/shims/python?

[AlJohri]

I'm using pyenv and vanilla virtualenvwrapper. @blueyed it does point to the shim python before I go into the virtualenv. The virtualenv adds it's own python to the beginning of the path.

[AlJohri]

I'll add some more debug output I can think of when I get to my laptop.

[AlJohri]

@blueyed I uninstalled pyenv and tried again. It seems to be an interaction with pyenv issue. Here is more specifically what's happening with pyenv:

no subshell

$ echo "$PATH"
bin:/Users/johria/.nodenv/shims:/Users/johria/.rbenv/shims:/Users/johria/.pyenv/shims:/usr/local/sbin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin

in subshell, same path

$ sh -c 'echo "$PATH"'
bin:/Users/johria/.nodenv/shims:/Users/johria/.rbenv/shims:/Users/johria/.pyenv/shims:/usr/local/sbin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin

in subshell with envdir, changed $PATH

$ envdir envs/prod sh -c 'echo "$PATH"'
/Users/johria/.pyenv/versions/3.6.1/bin:/usr/local/Cellar/pyenv/1.1.1/libexec:bin:/Users/johria/.nodenv/shims:/Users/johria/.rbenv/shims:/Users/johria/.pyenv/shims:/usr/local/sbin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin

pyenv is somehow adding two more folders on to the PATH when interacting with envdir. probably out of scope here but I'd appreciate any ideas for debugging further.

[AlJohri]

Another note: if envdir is installed inside the virtualenv, there are no issues.

[AlJohri]

I think the issue can be simplified down to:

$ which python
/Users/johria/.pyenv/shims/python
$ envdir envs/prod which python
/Users/johria/.pyenv/versions/3.6.1/bin/python

Still perplexing. I don't think any of my startup shell scripts are a culprit because if pyenv init ran properly it would go right back to the shims path.

[AlJohri]

It seems like I found the issue. Python itself is modifying the path when the python executable gets run.

$ echo "$PATH"
/Users/johria/.pyenv/shims:bin:/usr/local/sbin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin

$ python -c "import os; print(os.environ['PATH'])"
/Users/johria/.pyenv/versions/3.6.1/bin:/usr/local/Cellar/pyenv/1.1.1/libexec:/Users/johria/.pyenv/shims:bin:/usr/local/sbin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin

[AlJohri]

This only occurs with pyenv. With homebrew python:

$ echo "$PATH"
bin:/usr/local/sbin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin

$ python -c "import os; print(os.environ['PATH'])"
bin:/usr/local/sbin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin

No changes to the PATH. The pyenv wrapper shim is doing the addition to the PATH!

SO. In conclusion, when envdir is installed globally via pyenv, the pyenv shim is injecting variables on to the PATH. When inside a virtualenv, the shimmed envdir ends up changing the PATH so its no longer using the virtualenv but instead its using the global python.

Workaround: Install envdir using homebrew python or inside of your virtualenv. Do not install envdir via pyenv because it will change the PATH.

[floer32]

@AlJohri yeah, overall virtualenvs barely "exist," they are mostly about the modification to $PATH (see here for a good post about this). I recently learned more about this dealing with the interaction between pyenv and the wonderful (newish) Python package manager, poetry.

---

Anyways the envdir code of interest is https://github.com/jezdez/envdir/blob/master/envdir/runner.py#L79

To find a universal solution could be hard; for some people shell=True may be appropriate, others not. Judgment calls...

---

Tangential:

I'm beginning to think about writing a new utility, envdir in pure shell with just a dependency on env. It would have disadvantage vs Python envdir in not being compatible with non-sh/bash/zsh-like shells (like fish), but would possibly have the advantage of guaranteed consistency with the shell you call it from. (I really appreciate this repository, no doubt! Just thinking out loud)

[blueyed]

@hangtwenty Just a quick side offtopic note, I am using the following shell function to export envdirs (zsh):

envdir-export () {
    setopt extendedglob
    local envdir=$1 
    if ! [ -d "$envdir" ]
    then
        echo "$envdir is not a dir."
        return 1
    fi
    for i in $envdir/*(.)
    do
        if [[ "${i:t}" == *.* ]]
        then
            continue
        fi
        echo "Exporting $i:t"
        eval "export ${i:t}='$(<$i)'"
    done
}

I call this for example via autoenv (https://github.com/Tarrasch/zsh-autoenv) for projects, or manually on certain dirs to get extra parts of some config.