Env vars are ignored when using GUI in macOS

[Andre0991]

When using GUI Emacs in macOS,(shell-command-to-string "env") ~displays my env vars defined in .bashrc~, but not the ones in .bash_profile.

My config is here.

System Info :computer:

• OS: darwin
• Emacs: 25.3.1
• Spacemacs: 0.300.0
• Spacemacs branch: develop (rev. 1abce5ec1)
• Graphic display: t
• Distribution: spacemacs
• Editing style: hybrid
• Completion: ivy
• Layers:

  (clojure common-lisp emacs-lisp erc git github html ivy javascript latex markdown org osx plantuml shell-scripts scala spacemacs-layouts syntax-checking themes-megapack theming version-control
       (shell :variables shell-default-shell 'eshell)
       (elfeed :variables rmh-elfeed-org-files
               (list "~/Dropbox/backup/emacs/elfeed-feeds.org"))
       (c-c++ :variables c-c++-enable-clang-support t)
       (auto-completion :variables auto-completion-return-key-behavior 'complete auto-completion-tab-key-behavior 'cycle auto-completion-enable-help-tooltip nil auto-completion-enable-sort-by-usage t auto-completion-enable-snippets-in-popup nil :disabled-for latex)
       (spell-checking :variables spell-checking-enable-by-default nil enable-flyspell-auto-completion t)
       andre-common-lisp andre-elfeed andre-erc andre-eww andre-ivy nov)

• System configuration features: JPEG RSVG IMAGEMAGICK NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES

Also, I use bash from Homebrew. My default shell is /usr/local/bin/bash (correct according to env).

Related to https://github.com/syl20bnr/spacemacs/issues/10897.

[Andre0991]

Other interesting thing to notice is that Homebrew added /usr/local/bin to /etc/paths, which ends up added to PATH, but Spacemacs is not picking it up, so I can't use Homebrew commands from Emacs.

[Andre0991]

I just updated to the latest develop version (d3c594512d4e683edfa604053c424aa88c41b240), seems that even the vars from .bashrc are not being read.

What was happening is that I loaded a file that used exec-path-from-shell-copy-envs, so some of my vars were correctly when initialising it. As the vars seem to get cached, I didn't lose them when restarting Spacemacs with SPC q r. But I can confirm that it's not picking up any of my env vars.

[syl20bnr]

OSX layer was not responsible of importing env. vars, this was and still is done in spacemacs-bootstrap layer. We don't cache env. vars anymore, we fetch them all at startup time asynchronously.

Something prevents shell-command-to-string to work correctly in your environment. What is the value of shell-file-name ? What if you do M-! env RET ?

Could be Emacs 25 as well ? I'm on Emacs 26.

[Andre0991]

shell-file-name looks correct:

shell-file-name is a variable defined in ‘C source code’.
Its value is "/usr/local/bin/bash"
Original value was "/bin/bash"

env is incomplete too.

Good point, I'll update to Emacs 26.

[Andre0991]

Same issue with Emacs 26 :(

[anton-trunov]

I have the same issue. git bisect shows that 75f1d915a8be3c5cc9485341d4bbd871a2ca6e82 is what causes it.

[aaronjensen]

For me, env vars in .bashrc are loaded for me, but not those in .bash_profile. This is fine for me as I just source my bashrc from bash_profile. macOS has rather weird semantics in that it tends to run a login shell for every terminal so we get used to putting this in bash_profile that should probably actually be in bashrc. Of course, you could argue that you should already have the things in bash_profile in your env...

[Andre0991]

It's not getting vars from .bashrc here either, but it does pick up all vars if I start gui Emacs from terminal.

[fiveNinePlusR]

are you exporting your variables? env doesn't pick up the changes unless you export them.

export PATH="/foo/bar/biz:$PATH"

if you do this in a shell:

env | grep test # prints nothing
TEST="test"
echo $TEST # prints test
env | grep test # prints nothing. 
export TEST
env | grep test # prints TEST=test

I just did a test and emacs opens up the shell as an interactive shell on mine(zsh though) but one common strategy for dotfiles is to have the non-interactive shell config set the path and then have the interactive one source the non-interactive one. I am not sure which type bash is calling.

you can test it by doing this in your bashrc and bash_profile

if [ -z "$PS1" ]; then
    export TEST="non interactive"
else
    export TEST="interactive"
fi

and then testing inside ielm or m-: with (getenv "TEST")

[Andre0991]

Yes, I'm exporting them.

I just tested the snippet you posted. I put it in my .bashrc.

(getenv "TEST") returns interactive, as expected, when Emacs is started from terminal. (getenv "TEST") returns nil when Emacs is started from its icon.

How can I debug that?

[fiveNinePlusR]

where did you put that snippet? i would put it inside whatever the opposite file is and that is where you need to source the other file.

[Compro-Prasad]

What do you mean by opposite file? .bash_profile?

[AlexArrufat]

A quick solution for Zsh users. Create/open your ~/.zshenv file and write source ~/.zshrc. With this you will have the same config you have for interactive and non-interactive shells.

Following a better practice, I tend to put env-vars / initializers in ´~/.zshenv´ and the ones suited for an interactive shells in ~/.zshrc. I don't have issues in any apps with this approach.

[fiveNinePlusR]

What do you mean by opposite file? .bash_profile?

the interactive vs non-interactive file. i think it's .bashrc

[Andre0991]

@fiveNinePlusR I used .bashrc in my previous test. Putting it in .bash_profile doesn't help either, but I think this is expected in macOS (unless exec-path-from-shell is used).

@aaronjensen, are you using d12frosted/emacs-plus/emacs-plus from Homebrew?

To sum it up, the problem seems to be starting Emacs from GUI. It doesn't pick even bashrc in my case. Might be related to https://github.com/syl20bnr/spacemacs/issues/3268.

[aaronjensen]

@aaronjensen, are you using d12frosted/emacs-plus/emacs-plus from Homebrew?

Yes, with --HEAD though.

[Andre0991]

@aaronjensen, are you starting Emacs from terminal?

[aaronjensen]

@Andre0991 no

[chriskaukis]

I started getting this issue also today after updating. I was using Emacs Plus version, but also have the Emacs Mac versions installed. I usually use Emacs Plus because I like the sexier title bar. However, out of curiosity I tried Emacs Mac and I don't get the problem.

Summary: Problem occurs in Emacs Plus started from GUI, but not from command line. Emacs Mac seems to work regardless.

[mkleehammer]

I'm seeing this now with the Emacs Mac version from https://emacsformacosx.com

It may be intermittent. I can't get it to work now, but was pretty sure it did a couple of times. Someone mentioned "asynchronously" earlier - is it possible it is a timing issue and things are initializing before it completes?

[agzam]

as a lame workaround I'm doing this for now:

• add exec-path-from-shell to additional-pacakages
• set exec-path-from-shell-variables in user-init
• then manually call exec-path-from-shell-initialize (otherwise it doesn't work for me)

[mkleehammer]

I added (spacemacs/loadenv) to the bottom dotspacemacs/user-config and it seems to work. I don't see how since it is called last, but I wanted to work my way "backwards" in testing and it immediately started working.

[anton-trunov]

@mkleehammer Thanks for sharing!

Unfortunately, it didn't work for me since some of my layers rely on external utilities (e.g. ocaml layer relies on opam and merlin executables). So, instead I tried adding (spacemacs/loadenv) to dotspacemacs/user-init but I keep getting

(Spacemacs) Error in dotspacemacs/user-init: Symbol’s function definition is void: spacemacs/loadenv

So my workaround is cd ~/.emacs.d; git checkout d01658ec6.

[syl20bnr]

@anton-trunov can you test with a quick patch of the spacemacs code there ?

https://github.com/syl20bnr/spacemacs/blob/develop/layers/+distributions/spacemacs-bootstrap/funcs.el#L26-L27

Replace the if expression with:

      (let ((shell-command-switch "-ic"))
        (if dotspacemacs-import-env-vars-shell-file-name
            (let ((shell-file-name
                   dotspacemacs-import-env-vars-shell-file-name))
              (split-string (shell-command-to-string "env") "\n"))
          (split-string (shell-command-to-string "env") "\n"))))

the idea is to pass -i when invoking the shell.

[syl20bnr]

@anton-trunov you can try by just evaluating the code above, no need to patch the code I believe. What does the let bindings above returns for you ?

[anton-trunov]

@syl20bnr Thanks a bunch for looking into this issue! It's much appreciated. I'm working on a layer for the Lean theorem prover and looking forward to be able to contribute back to the Spacemacs community.

Here is the output for M-x eval-expression before applying the patch on current develop branch (6b4a79d3981419526a84956161b0e639c1a3d5a9):

("PWD=/Users/anton" "RUST_SRC_PATH=/Users/anton/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/src" "PKG_CONFIG_PATH=/usr/local/opt/libffi/lib/pkgconfig:/usr/local/opt/libffi/lib/pkgconfig:" "OLDPWD=/Users/anton" "SHLVL=0" "DISPLAY=192.168.1.132" "TERM=xterm-256color" "LANG=ru_RU.UTF-8" "XPC_FLAGS=0x0" "USER=anton" "XPC_SERVICE_NAME=org.gnu.Emacs.17160" "LOGNAME=anton" "PATH=/Users/anton/.opam/coq.8.8.0/bin:/usr/local/smlnj/bin:/usr/local/opt/coreutils/libexec/gnubin:/Users/anton/.cargo/bin/:/Users/anton/prj/coq/coq-bench:/Users/anton/prj/coq/coq-tools:/Users/anton/.cabal/bin:/Users/anton/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/Users/anton/.cargo/bin/:/Users/anton/prj/coq/coq-bench:/Users/anton/prj/coq/coq-tools:/Users/anton/.cabal/bin:/Users/anton/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/opt/fzf/bin" "SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.r8O0BAPgJi/Listeners" "Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.YaTy1pYZJi/Render" "HOME=/Users/anton" "SHELL=/bin/zsh" "__CF_USER_TEXT_ENCODING=0x1F6:0x0:0x0" "TMPDIR=/var/folders/6c/_7dzfcbd5q30wyqhppf7gf800000gp/T/" "LC_ALL=ru_RU.UTF-8" "ZSH=/Users/anton/.oh-my-zsh" "PAGER=less" "LESS=-R" "LSCOLORS=Gxfxcxdxbxegedabagacad" "OCAML_TOPLEVEL_PATH=/Users/anton/.opam/coq.8.8.0/lib/toplevel" "PERL5LIB=/Users/anton/.opam/coq.8.8.0/lib/perl5:" "MANPATH=:/Users/anton/.opam/coq.8.8.0/man" "OPAMUTF8MSGS=1" "CAML_LD_LIBRARY_PATH=/Users/anton/.opam/coq.8.8.0/lib/stublibs" "_=/usr/local/opt/coreutils/libexec/gnubin/env" "")

Alas, applying the patch it didn't work. Here is the log in `Messages:

Loading /Users/anton/.spacemacs...done
(Spacemacs) Warning: Cannot find "opam" or "merlin" executable. The ocaml layer won't work properly.
Starting new Ispell process ispell with default dictionary...
Error enabling Flyspell mode:
(Searching for program No such file or directory ispell)
Spacemacs is ready.
Starting new Ispell process ispell with default dictionary...
Error enabling Flyspell mode:
(Searching for program No such file or directory ispell)
Loading /Users/anton/.emacs.d/.cache/recentf...done
Skipping check for new version (reason: dotfile)
.emacs.d/layers/+spacemacs/spacemacs-purpose/local/spacemacs-purpose-popwin/spacemacs-purpose-popwin.el: Obsolete name arg "pupo" to constructor purpose-conf

The opam and merlin are in PATH (and ispell is too):

$ which opam
/usr/local/bin/opam
$ which ocamlmerlin
/Users/anton/.opam/coq.8.8.0/bin/ocamlmerlin

Just in case, here is the diff for your patch that I applied:

-      (if dotspacemacs-import-env-vars-shell-file-name
-          (let ((shell-file-name
-                 dotspacemacs-import-env-vars-shell-file-name))
-            (split-string (shell-command-to-string "env") "\n"))
-        (split-string (shell-command-to-string "env") "\n")))
+      (let ((shell-command-switch "-ic"))
+        (if dotspacemacs-import-env-vars-shell-file-name
+            (let ((shell-file-name
+                   dotspacemacs-import-env-vars-shell-file-name))
+              (split-string (shell-command-to-string "env") "\n"))
+          (split-string (shell-command-to-string "env") "\n"))))

[dsedivec]

@anton-trunov can you test with a quick patch of the spacemacs code there ?

Thanks @syl20bnr, I think you're on the right track here. I'm on macOS, having same problems described here. -i fixes it for me:

ELISP> (print (shell-command-to-string "env"))

"[...]
PATH=/usr/bin:/bin:/usr/sbin:/sbin
"
[...]

ELISP> (let ((shell-command-switch "-ic")) (print (shell-command-to-string "env")))

"bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
[WTF?  Anyway, later on...]
PATH=/Users/dale/bin:/Users/dale/.go/bin:/Users/dale/.cabal/bin:[and the rest of my ridiculous path]
[...]
"

As further confirmation, it looks like exec-path-from-shell uses -l -i (conditionally-ish). Apparently I am defining my variables in the "wrong" init file, as are many others I expect.

(I'm interested to find why exec-path-from-shell was dropped, actually. I'll search the pull requests, unless someone has an issue/pull request number handy.)

[fiveNinePlusR]

use -lc instead of -ic and it should load up the rc file.

[dsedivec]

Maybe not quite apropos this thread, but since I'm already here: Looks like spacemacs-env-vars-file is not being used? Getting the environment asynchronously in spacemacs/loadenv causes some problems here for me here on macOS. Might have to do with restoring open buffers on startup, such as ones that enable flyspell-mode. Excerpt from *Messages* with --debug-init:

[...]
Loading /Users/dale/.emacs.d/.cache/spacemacs-buffer.el (source)...done
Starting a server...
Starting new Ispell process ispell with default dictionary...
Error enabling Flyspell mode:
(Searching for program No such file or directory ispell)
Spacemacs is ready.
[...then *much* later on...]
(Spacemacs) Imported environment variables:

Not sure if async environment variable loading is a good idea?

[fiveNinePlusR]

I don't think that it's worth making async personally... too many other issues can arise from not having the proper environment variables setup to make it worth the small gains in loading speed.

[anton-trunov]

use -lc instead of -ic and it should load up the rc file.

That didn't help in my case. (I'm not sure if it was supposed to.)

It looks like @dsedivec has pinpointed the problem.

[syl20bnr]

OK so we are close. We can remove the async fetch as even a hook may not be enough but I will re-introduce the caching then. In all cases SPC f e R will always update the cache. I'll also add a dotfile variable for people wanting to opt-out the caching of env. vars. as well as writing a message in messages buffer about the fact that env. vars. are being loaded from a file and SPC f e R can be used to reload them, or even just SPC SPC spacemacs/update-env or something.

I really want to optimize starting time for Spacemacs, I got too much feedback about it being bloated because it starts slower than doom-emacs. I don't like that :-)

[syl20bnr]

I just pushed a rework of the management of environment variables. Getting env. vars from shell is not reliable and so this new approach goes away from it. It is like an hybrid of the different solutions we tried before.

Spacemacs uses its own .env file stored in ~/.emacs.d/private/spacemacs.env or ~/.spacemacs.d/spacemacs.env. If the file does not exist then Spacemacs tries to initialize it from user shell and write a message in *messages* about it.

To open the file spacemacs.env for edition use SPC f e e, SPC f e E to reload it and SPC f e C-e to re-initialize it.

I'm going to sleep a bit, I think that I'll add the location ~/.spacemacs.env also tomorrow.

[Blaisorblade]

@syl20bnr The new approach sounds somewhat reasonable, tho it needs at least to add -l to bash options (https://github.com/syl20bnr/spacemacs/commit/6220ace290b247265de8fac66bd9e84bef388326#r29480946). Calling env is fine, but I'd keep everything else as close to exec-path-from-shell as possible.

But I'd want a way to keep using exec-path-from-shell, because my environment changes more often than I want to update my Emacs config — quite probably, a documented toggle, since this affects the bootstrap phase.

[Blaisorblade]

For the others who've been debugging this: FWIW, #10951 has an analysis of part of the problems with the shell — usually you need -ilc to load all the init files, -lc should in fact be fine if env. vars are set in ~/.profile, which they should.

@dsedivec Regarding

"bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
[WTF?  Anyway, later on...]

-i means interactive and expects a terminal, which apparently that shell didn't get. Reading the code from exec-path-from-shell I can't tell if it does something else — it only bypasses process-file. I guess that might make ~/.bashrc fail before setting the environment, tho a scenario like that probably can't be helped other than by moving environment settings in ~/.profile or ~/.bash_profile, where they arguably belong.

[syl20bnr]

It will be possible to use exec-path-from-shell in the final implementation. I will introduce the function user-env in the dotfile where by default it reads the spacemacs.env file but users will be able to do anything they want in this function, like using exec-path-from-shell or simply using setenv and set exec-path themselves.

SPC f e E will call dotspacemacs/user-env so anyone will be able to hot update their environment variables. SPC f e R will also call this function.

SPC f e e will open spacemacs.env if it exists or go directly to the dotspacemacs/user-env function otherwise. I'll add a header to the spacemacs.env file to explain what it is etc...

Basically with this I expect to make environment variables a first class citizen of spacemacs configuration instead of a magical obfucasted thing.

Should make things more explicit and give enough flexibility and clarity to the users.

"Paolo G. Giarrusso" notifications@github.com writes:

@syl20bnr The new approach sounds somewhat reasonable, tho it needs at least to add -l to bash options (https://github.com/syl20bnr/spacemacs/commit/6220ace290b247265de8fac66bd9e84bef388326#r29480946). Calling env is fine, but I'd keep everything else as close to exec-path-from-shell as possible.

But I'd want a way to keep using exec-path-from-shell, because my environment changes more often than I want to update my Emacs config — quite probably, a documented toggle, since this affects the bootstrap phase.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/syl20bnr/spacemacs/issues/10906#issuecomment-399882612

[panchoh]

Hi!

I just wanted to add that this caching of the env vars is making my life difficult, since there is an essential env var in my setup whose value changes every time I reboot: SSH_AUTH_SOCK, which is the way git is able to communicate with my ssh-agent. In short, unless I force a refresh of the environment, everytime I tell magit to pull or push, it will ask me for a passphrase, instead of leveraging my ssh-agent.

I reckon that is is a rather common setup, and I wonder why are there not more people mentioning it here...

(Running develop on Arch Linux).

Thanks a bunch! Spacemacs is truly awesome!

[benreyn]

Im having the same problem as @panchoh . Any word on how this could be resolved or worked around?

[panchoh]

Im having the same problem as @panchoh . Any word on how this could be resolved or worked around?

I simply remove ~/.spacemacs.d/spacemacs.env before launching emacs.

[dsedivec]

Maybe just put (spacemacs/init-env t) in your dotspacemacs/user-init function? Seems like that should do the trick, though I haven't tested it.

[panchoh]

Maybe just put (spacemacs/init-env t) in your dotspacemacs/user-init function? Seems like that should do the trick, though I haven't tested it.

Thanks, @dsedivec, that worked like a charm!

[syl20bnr]

I pushed the modification to the env vars management described above. See commit message and its content for more info: https://github.com/syl20bnr/spacemacs/commit/a013d8687

[panchoh]

I reckon that GPG_AGENT_INFO should probably be blacklisted, too, @syl20bnr. And possibly SSH_AGENT_PID.

[mkleehammer]

I really like the new design, but something happened in this latest commit a013d8687 and on macOS I am no longer getting entries from /etc/paths, even after running the new force init function.

The path in my .spacemacs.env file does not include /usr/local/bin et. al. but running "env" on the shell, and running bash -ic env, generates the PATH with the /usr/local/bin entries.

I assume something changed and it is no longer using a login shell or something, though I'm not seeing it in the latest commit.

I've manually updated the path in my env file for now, but what can I do to help with this?

[benreyn]

I reckon that GPG_AGENT_INFO should probably be blacklisted, too, @syl20bnr. And possibly SSH_AGENT_PID.

I wonder if a configurable variable in the dotfile would be a good idea?

[mkleehammer]

Hmmm... As I mentioned in my previous comment, bash -ic env includes entries from /etc/path. However, when I execute the following, the result does not include anything from /etc/path:

(let ((shell-command-switch "-ic"))
  (message "%s" (shell-command-to-string "env")))

It's almost as if shell-command-switch isn't getting used, but shell-command-to-string certainly looks right. (Though I'm not an expert in elisp.). Am I missing something obvious or is shell-command-switch getting ignored or overwritten somehow?

My shell-file-name is "/bin/bash"

[Blaisorblade]

@mkleehammer Spacemacs will need to call bash -ilc env, not bash -ic env, and I already pointed out on the commit that this fix was needed (https://github.com/syl20bnr/spacemacs/commit/6220ace290b247265de8fac66bd9e84bef388326#r29480946); I'm making a PR. You can test the change locally.

Your experiments to the contrary are misleading for one reason:

The path in my .spacemacs.env file does not include /usr/local/bin et. al. but running "env" on the shell, and running bash -ic env, generates the PATH with the /usr/local/bin entries.

The shell you're running the commands into has already the correct PATH, so bash -ic env shows it even though it doesn't account for /etc/paths. The PATH is only initialized by login shells that is, ones started with -l; non-login shells (such as bash -i) assume the parent processes have already done that initialization.

On OS X, /etc/paths is only read by as part of PATH initialization by login shells.

EDIT: the actual change should be https://github.com/syl20bnr/spacemacs/pull/10992 (yet untested).

[mkleehammer]

@Blaisorblade Got it - thanks for the clarification. I forgot env is just going to print what is already set.

There seems to be one more thing - the exec-path is not getting updated.

I applied the patch, deleted the .spacemacs.env file, and restarted. The env file was recreated and did include the /etc/path entries. However, exec-path did not match the PATH from env file. Restarting successfully loaded the cached env file, but again the exec-path did not match PATH in the env file.

Is exec-path getting set too soon?

[Blaisorblade]

@mkleehammer Not sure myself on exec-path, you'll need an Elisp expert for that.

[mkleehammer]

Is this patch supposed to work on Windows or are more updates coming? After updating Windows, including syncing the Spacemacs init template, I get this in my env file:

Copyright (c) 2009 Microsoft Corporation.  All rights reserved.
Microsoft Windows [Version 6.1.7601]
c:\bin\emacs\bin>
warning: extra args ignored after '-ic'

If I take out -ic and restart, I get this:

Copyright (c) 2009 Microsoft Corporation.  All rights reserved.
Microsoft Windows [Version 6.1.7601]
c:\bin\emacs\bin>
warning: extra args ignored after 'c:\bin\emacs\libexec\emacs\26.1\x86_64-w64-mingw32\cmdproxy.exe'

I don't exactly remember, but I think this is the official Emacs 26.1 for Windows.