Environment Broken

[elemakil]

I'm using an environment with a postactivate hook given by virtualenvwrapper. This hook sets some environment variables (e.g. PYTHONPATH) required for executing the code I am developing.

I am using a .dir-locals.el file at the base on my project with the following config:

((python-mode
  (eval ignore-errors
        "Setup python environment variables."
        (make-local-variable 'process-environment)
        (setq process-environment
              (append process-environment
                      '("PYTHONPATH=/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages"
                        "VIRTUALENVWRAPPER_ENV_BIN_DIR=bin"
                        "VIRTUALENVWRAPPER_HOOK_DIR=/Users/elemakil/.virtualenvs"
                        "VIRTUALENVWRAPPER_PYTHON=/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python"
                        "VIRTUALENVWRAPPER_SCRIPT=/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/virtualenvwrapper.sh"
                        "VIRTUALENVWRAPPER_VIRTUALENV=virtualenv"))))))

As expected, this modifes the variables relevant for the hook procedure (VIRTUALENVWRAPPER_*) only for the relevant buffers. However, when loading the environment using pyvenv (pyvnev-workon), these variables are not used.

For debugging, I've put an echo statement into the pyvenv-run-virtualenvwrapper-hook function [L 370-2]:

(call-process-shell-command
               (format ". '%s' ; echo \"PYTHONPATH = '$PYTHONPATH'\" >> /Users/elemakil/foo ; echo \"VIRTUALENVWRAPPER_HOOK_DIR = '$VIRTUALENVWRAPPER_HOOK_DIR\" >> /Users/elemakil/foo ; echo ; echo =-=-= ; python -c \"import os, json ; print(json.dumps(dict(os.environ)))\""
                       tmpfile)
               nil t nil)

Both variables are empty. Consequently, an interactive session started from within emacs will fail to work correctly since the code in the hook is not executed.

Please advise!

[jorgenschaefer]

Hello, and thanks for the question! I am afraid Emacs has very weird behavior when it comes to buffer-local values of process-environment, as the inheritance of buffer-local variables to other buffers is nearly impossible to get right – I do not think you will have much luck with that (I tried). For this reason, virtualenvs in Emacs are global.

[elemakil]

Hmm.. that is annoying but I guess no way around?

Another problem I noticed when making it no longer buffer local is that there's a json-read: JSON readtable error when the hook script prints output to stdout. The content of the output buffer is:

~/PROJECT/source/main ~/PROJECT/source/main
~/PROJECT/source/main

=-=-=
{"VIRTUALENVWRAPPER_ENV_BIN_DIR": "bin", "LIBPATH": "/opt/local/libexec/root6/lib/root", "LOGNAME": "elemakil", "USER": "elemakil", "PATH": "/opt/local/libexec/root6/bin:/Users/elemakil/.virtualenvs/roo/bin:/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/:/opt/local/bin/:/Users/elemakil/.gem/ruby/2.0/bin/:/Users/elemakil/.local/bin/:/Users/elemakil/.usr/bin/:/Users/elemakil/Documents/Dots/.bin/:/usr/local/opt/coreutils/libexec/gnubin/:/usr/local/bin/:/usr/bin/:/bin/:/usr/sbin/:/sbin/:/opt/X11/bin/:/Library/TeX/texbin/:/Users/elemakil/Documents/Dots/fzf/.fzf/bin/:/Users/elemakil/Documents/Dots/zsh/zsh/plugins/git-extra-commands/:/usr/local/Cellar/emacs-mac/emacs-24.5-z-mac-5.8/libexec/emacs/24.5/x86_64-apple-darwin14.4.0/:/Users/elemakil/PROJECT/bin", "HOME": "/Users/elemakil", "VIRTUALENVWRAPPER_SCRIPT": "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/virtualenvwrapper.sh", "DISPLAY": "/private/tmp/com.apple.launchd.1aDnKrFnbk/org.macosforge.xquartz:0", "TERM": "dumb", "SHELL": "/usr/local/bin/zsh", "VIRTUALENVWRAPPER_PYTHON": "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python", "SHLVL": "1", "XPC_FLAGS": "0x0", "LD_LIBRARY_PATH": "/opt/local/libexec/root6/lib/root", "TMPDIR": "/var/folders/jj/4cnxg9m546dgb4mxdn5b5x7c0000gn/T/", "MANPATH": "/opt/local/libexec/root6/share::/Users/elemakil/Documents/Dots/fzf/.fzf/man", "SSH_AUTH_SOCK": "/private/tmp/com.apple.launchd.HkThE7oKre/Listeners", "PYTHONPATH": "/opt/local/libexec/root6/lib/root:/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages:/Users/elemakil/PROJECT/source", "XPC_SERVICE_NAME": "0", "VIRTUAL_ENV": "/Users/elemakil/.virtualenvs/roo", "DYLD_LIBRARY_PATH": "/opt/local/libexec/root6/lib/root", "Apple_PubSub_Socket_Render": "/private/tmp/com.apple.launchd.D5BjQ5CVnv/Render", "_": "/Users/elemakil/.virtualenvs/roo/bin/python", "VIRTUALENVWRAPPER_HOOK_DIR": "/Users/elemakil/.virtualenvs", "OLDPWD": "/opt/local", "ROOTSYS": "/opt/local", "__CF_USER_TEXT_ENCODING": "0x1F5:0x0:0x0", "PWD": "/Users/elemakil/PROJECT/source/main", "VIRTUALENVWRAPPER_VIRTUALENV": "virtualenv", "SHLIB_PATH": "/opt/local/libexec/root6/lib/root"}

I obtained this by inserting

(message (buffer-substring-no-properties (point-min) (point-max)))

just after line 374.

Edit: This is actually really surprising: If I put

(message "|%s|" (buffer-substring-no-properties (point) (point-max)))

just after the

(when (and (not (re-search-forward "ImportError: No module named virtualenvwrapper" nil t))
                 (re-search-forward "\n=-=-=\n" nil t))

the printed message corresponds to exactly one json cell (with a subsequent newline).

If I paste this cell into a new buffer and manually call json-read there is no error...

[jorgenschaefer]

Hmm.. that is annoying but I guess no way around?

Nope. I wish there was. Emacs would need project-local variables as opposed to buffer-local ones to make process-environment have the right values in new buffers spawned for subshells, too. No such facility exists. Everything else would involve elaborate hacks that will regularly fail in unexpected ways, too. I'm open to ideas, though! :-D

Another problem I noticed when making it no longer buffer local is that there's a json-read: JSON readtable error when the hook script prints output to stdout.

Could you post the full tracback?

[elemakil]

Haha! I figured it out. In the master version, if additional output prior to the =-=-= marker is discovered, a help buffer is created that lists the output. However, this results in a change of the buffer onto which json-read is applied. Reversing the order or printing and parsing solves the problem:

(when (and (not (re-search-forward "ImportError: No module named virtualenvwrapper" nil t))
                 (re-search-forward "\n=-=-=\n" nil t))
        ;; Collect the stdout output from the hook command
        (let ((output (buffer-substring (point-min)
                                        (match-beginning 0))))
          ;; parse the json
          (dolist (binding (json-read))
            (let ((env (format "%s=%s" (car binding) (cdr binding))))
              (when (not (member env process-environment))
                (setq process-environment (cons env process-environment))))
            (when (eq (car binding) 'PATH)
              (setq exec-path (split-string (cdr binding) ":"))))
          ;; assuming that there was output from the hook command, show it to the user
          (when (> (length output) 0)
            (with-help-window "*Virtualenvwrapper Hook Output*"
              (with-current-buffer "*Virtualenvwrapper Hook Output*"
                (let ((inhibit-read-only t))
                  (erase-buffer)
                  (insert
                   (format
                    "Output from the virtualenvwrapper hook %s:\n\n"
                    hook)
                   output)))))))

[jorgenschaefer]

Wow, nice catch!