Asynchronous direnv calls

[Drainful]

I have hacked Envrc (gist) to call direnv asynchronously (with make-process and a sentinel) because as a Guix user using Envrc to establish a guix environment can block for quite a while as files are downloaded and binaries are built. I imagine the same problem happens when using direnv to establish a nix shell environment. The simplest solution is to manually establish the environment once to let the software be installed before using Envrc, but this doesn't feel great as it could be done automatically.

Should I polish this feature so that it could be included, or do you think it doesn't fit with the project?

[purcell]

Hi! Thanks for starting the conversation about this. Asynchronous updates are somewhat on my radar. You would think it would be an issue for me as an increasingly enthusiastic Nix user (I'm sure Guix is great too!), but the problem has largely been solved for me by lorri. I therefore wonder if the best solution overall would be to make something like that for Guix, because I imagine that Nix and Guix are the main two situations in which direnv evaluations might end up taking a long time.

In terms of envrc.el I'm wary of the extra complexity of asynchrony, but I might consider explicitly handling it at some point. The two key worries about asynchronous updates are:

1. Multiplexing async updates triggered by many buffers and applying to many buffers
2. Having a period of time in which buffers' "direnv state" is indeterminate.

Generally speaking, I'd like to start calling direnv more aggressively (e.g. when new buffers are created, even when "inside" a known direnv), and this would lead to propagating changes asynchronously back to other buffers "in" that direnv. So that relates to point 2, but at least the initial state of each buffer would always have a valid direnv result due to the blocking calls. Point 1 is manageable, but just adds a lot of code.

So I guess overall I'd like to better understand cases where people use .envrc files that can take a long time to evaluate, because that seems somewhat antithetical to direnv. use_nix was arguably a bad citizen in this sense, and perhaps the same applies to Guix.

Another related thing I need to do is tackle "reloading" a direnv by re-using the existing var values when re-invoking direnv, so that it can see and take advantage of DIRENV_WATCHES/DIRENV_MTIME to quickly provide cached responses if appropriate.

Any thoughts are appreciated: this is very much a 0.1.x release, and I plan to iterate on it as time allows and as I and other users get to know the pain points, like this issue.

[Drainful]

I think the optimal solution would be to make some kind of lorri for guix, but if we were to go down the asynchronous envrc route then I don't think the indeterminate state issue would be insurmountable. Even with Lorri you could access your project before the Lorri daemon has first initialized and be in a state that could be considered invalid.

If you are using Envrc and you make some change to your .envrc that would result in a long direnv refresh, and envrc-reload is called (manually or automatically) then Envrc doesn't need to modify your environment variables until the asynchronous direnv call finishes, leaving you in a valid state while you wait. If more envrc-reload calls are made with the same parent env-dir as a current reload then they could be ignored.

All things considered I think my problem could be more cleanly solved from the Guix side. Regardless, It might be reasonable to develop this feature anyway since from the perspective of someone using direnv from a shell where you can just send a long running process to the background, having emacs lock up with no recourse could be jarring in comparison, even for a short duration.

It might not be worth the effort or the extra complexity though. I'd be happy with the project either way.

[purcell]

Yeah, agree. I'll definitely have a longer think about this.

[Mic92]

I tried hacking on emacs-direnv to support this feature in my own fork https://github.com/Mic92/emacs-direnv/commit/f4f3dbb085c926c451f3515d846606420c4d5a2d it sort of work but there are some bugs described here: https://discourse.nixos.org/t/emacs-direnv-help-needed-to-make-it-non-blocking/8595 @DamienCassou pointed me to this project. lorri does not work for me because I have a git checkout of nixpkgs in my NIX_PATH which makes lorri eat my cpu whenever I try to checkout a different branch. Also it cannot handle flakes yet. Both is addressed by https://github.com/nix-community/nix-direnv

[purcell]

@mic92 Yeah, envrc is still blocking, ultimately, though it should re-evaluate less than direnv.el. I also use nix-direnv instead of direnv's builtin use_nix, though it can still lead to blocking evaluations in some cases.

[Mic92]

Yes. nix-direnv will block if files needs to be re-evaluated or packages need to be downloaded.

[dvzubarev]

Sorry for chiming in, I was using this hack to avoid blocking on slow nix re-evaluations. I'm not a fan of starting another daemon, so I use a lorri watch sub-command with --once flag. This command is invoked asynchrinously after saving shell.nix or default.nix. When the lorri watch is done (envrc-reload) is launched. I'd prefer to use nix-direnv or something simpler than lorri If it provides a way to start re-evaluation from cli.

(defun my-update-environment ()
  (interactive)
  (envrc-reload)
  ;; (my-restart-ycmd)
  )

(defun my-run-lorri-watch-sentinel (process event)
  (if (equal event "finished\n")
      (my-update-environment)
    (message "Process %s event %s" process event)))

(defun my-run-lorri-on-shell-nix-change ()
  (interactive)
  (when (projectile-project-p)
    (let ((process-connection-type nil))  ; use a pipe
      (start-file-process "lorri-watch"
                          "*lorri*"
                          "lorri" "watch" "--once")
      (set-process-sentinel (get-process "lorri-watch") 'my-run-lorri-watch-sentinel))))

(defvar my-lorri-watch-files '("default.nix" "shell.nix"))

(add-hook 'nix-mode-hook
          (defun enable-autoreload-for-nix-shell ()
            (when (and (buffer-file-name)
                       (member (file-name-nondirectory (buffer-file-name))
                               my-lorri-watch-files))
              (add-hook 'after-save-hook 'my-run-lorri-on-shell-nix-change t t))))

[Mic92]

Thanks for sharing. This is a nice approach unfortunately I need a solution for flakes now as well and the author of lorri does not like flakes, so we won't see this beeing implemented soon. However I think using https://eradman.com/entrproject/ with direnv could solve this: echo .envrc default.nix shell.nix flake.nix flake.lock | entr direnv exec . true as well.

[Mic92]

What does the rest of your configurations looks like? How do you disable envrc by default otherwise?

[dvzubarev]

What does the rest of your configurations looks like?

(use-package envrc
:config
(envrc-global-mode))

How do you disable envrc by default otherwise?

Not sure what do you mean. Auto envrc-reload is called after editing shell.nix or default.nix in the root of your project.

direnv exec . true

Thank you for the tip! I replaced lorri watch --once with this command:

(defun my-update-environment ()
  (interactive)
  (envrc-reload)
  (message "envrc was reloaded.")
  ;; (my-restart-ycmd)
  )

(defun my-run-direnv-exec-watch-sentinel (process event)
  (if (equal event "finished\n")
      (my-update-environment)
    (message "Process %s event %s" process event)))

(defun my-run-direnv-exec-on-shell-nix-change ()
  (interactive)
  (when (projectile-project-p)
    (let ((process-connection-type nil))  ; use a pipe
      (start-file-process "direnv-exec"
                          "*direnv-exec*"
                          "direnv" "exec" "." "true")
      (set-process-sentinel (get-process "direnv-exec") 'my-run-direnv-exec-watch-sentinel))))

(defvar my-nix-project-watch-files '("default.nix" "shell.nix" "flake.nix"))

(add-hook 'nix-mode-hook
          (defun enable-autoreload-for-nix-shell ()
            (when (and (buffer-file-name)
                       (projectile-project-p)
                       (member (file-relative-name buffer-file-name (projectile-project-root))
                               my-nix-project-watch-files))
              (add-hook 'after-save-hook 'my-run-direnv-exec-on-shell-nix-change t t))))

It has limitations that it won't update an environment if you edit a file that is imported from shell.nix or when you regenerate flake.lock.

[Mic92]

I think I mainly misunderstood what you did. I thought you would only load direnv on certain events asynchronously, but you only handle reloads this way. However is there maybe a project tile hook one could use to load direnv asynchronously on the first run?

[dvzubarev]

It seems that there are only hooks that trigger when you use projectile-switch-project. You may try (add-hook 'projectile-after-switch-project-hook #'my-run-direnv-exec-on-shell-nix-change). But hook won't run if you open a file of a project via e.g. find-file.

[Mic92]

would emacs ./foo use find-file rather than the projectile hook?

[Mic92]

My fork is now asynchronous: https://github.com/Mic92/envrc/tree/async

This is how to use it in doom-emacs:

(package! envrc
  :pin "0c220b033b627fb58fdeaaaa12ae868eb775ef6c"
  :recipe (:host github :repo "Mic92/envrc"))

If someone wants to upstream this feature, feel free to take my code.

[ParetoOptimalDev]

I'll have to try your fork out!

About Lorri mentioned earlier in this thread...

I haven't tried in some time, but I found Lorri to be very unreliable or at least not work as direnv does and had to abandon it.

[psionic-k]

My fork is now asynchronous: https://github.com/Mic92/envrc/tree/async

From what I read, the implementation would allow later minor modes / hooks to run that might expect the loaded environment to be available.

The reason we are supposed to add the mode hook after other minor modes is so that they will not see the incomplete environment (because of LIFO hook ordering).

While we don't want Emacs to block, I don't think loading the environment out of order is going to solve more problems than it causes for situations such as project provided language servers or environment settings for them.

I'm not sure if it's possible to block just a single buffer. To implement it without any Emacs integration will probably mean stashing the hooks for a new buffer, replacing the buffer contents with some non-blocking loading indication, and then unstashing and restarting hooks after the environment is finished. How / how cleanly it can be done is the main issue on my mind.

[Mic92]

Well than it's going to be a long-term fork. I cannot have emacs blocking when I just want to look at a file. I rather restart my lsp if needed.

[psionic-k]

Blocking isn't accurate. I think I have a better proposal, but first what I meant was to hijack the remainder of the mode switch and then run it after direnv finishes. The file would be visible and interactive, but with almost no minor modes active.

We can do the same thing using two hooks instead of one and no hacking. The first hook on the major mode would put a function into the envrc hook. The second hook to load the minor mode would go off for both updates and asynchronous initialization. This solution depends on asynchronously loading the direnv while the buffer major mode is already finished.

I think we can write a function or macro to create such a chain-loading hook function. Like other hooks, it would execute immediately if the direnv is set up or be called after direnv finishes. Having a hook would give some minor modes a chance to both initialize late and to re-initialize on any direnv update.

Minor modes that are stateless, using getenv on every command, don't care about the environment changing out from under them. Only minor modes that derive state from the direnv and hold onto it in elisp need to be notified of environment updates. It's not super common, and by using a chain-hook generator, the user still writes the hook to begin on the major mode hook.

We usually only want a language server for certain major modes. We need a direnv hook to actually load the minor mode for such a case. Still, it's wrong to load the minor mode on every direnv without looking at the major mode. The first hook handles the major mode decision while the second hook handles the late and re-initialization.

So the solution is to use the major mode hook to set up a direnv hook to load minor modes that depend on direnv.

[ParetoOptimalDev]

I'm not sure if others might find it useful and it might not be appropriate for the implementations discussed but... maybe my idea could be useful.

I'm a very happy detached.el user and my ideal envrc-mode would run the typical blocking call using (perhaps) detached-shell-command and then after that's done running do the usual hooks after to update the environment.

This gives:

• notifications of when it's done if you've set them up
• a log of the build "session" for something like nix develop or whatever you are using direnv for
• async basically, or at least it doesn't block anything

[bbigras]

Anyone tried the ideas from the 2 last comments?

[purcell]

Good news, I think... first a couple of comments.

I think there's a plausible argument that direnv is going to block work even in a terminal when an .envrc can take an indeterminate amount of time to evaluate. Nix is the outlier in causing such issues, and that's why there's also lorri and sorri, either of which can theoretically completely solve the issue at hand. I haven't used either, and I don't know if either supports Flakes these days. It seems to me like solving this issue optimally in Emacs is equivalent to writing such a thing, and the likes of lorri are quite complex.

I like that @Mic92's changes are pretty minimal, but it seems like the result will still be unpredictable use of outdated environments during mode startup and other times, and that doesn't seem a good default.

Anyway, I'd noticed that I could usually hit C-g to cancel blocking direnv invocations, but of course that interrupt bubbles up to interfere with mode hooks etc., and envrc.el might immediately try the same invocation again. So I've tried a different approach in #54, see the comments there. Net result is that everything remains synchronous, but interruption is actively supported and does something reasonable. For me, this feels like a good balance of practicality and simplicity, keen to hear thoughts.

[purcell]

(Also CC-ing @sellout here, who opened #53 about the same topic.)

[Mic92]

Yeah breaking emacs (for example syntax highlighting) by hitting Ctrl-g was indeed an annoyance. So far I have not seen any downsides for the async variant for my personal usage. It's usually projects, where I just want to open a single file and where I don't even wait for .envrc to load when changing to it, where emacs would block. So it works great for me. I don't see how lorri would solve this for emacs: If it does not block, how does it return the right information when envrc-mode loads it?

[purcell]

lorri and sorri always return cached results, they never re-evaluate synchronously: they run a background process which does the synchronous bit.

[purcell]

(Just to be clear, with the new change, C-g is basically well-behaved, because it won't bubble up beyond envrc.el's code to break stuff like font lock.)

[zeorin]

Hmmm, I use https://github.com/nix-community/nix-direnv (instead of Lorri) and in my regular terminal emulator + shell (Kitty + fish) that works great and the use nix is cached. Similarly, when I open a file in Emacs in a folder that is affected by an .envrc file that uses use nix it's also fast.

But when I open a vterm buffer in Emacs in a dir that has such an .envrc it hangs for quite a bit (at least, it does the first time that I do that for the current Emacs daemon process—subsequently it's fast), presumably because it's setting up the nix shell, but I don't really understand why that would be taking so long (several minutes) when the nix shell is already cached.

I have nix-direnv installed via https://nix-community.github.io/home-manager/, which adds it to my ~/.config/direnv setup for me, but even when I use the .envrc installation method it still takes just as long (again, just the first time for current process).

Maybe this is unrelated to the OP's issue and it's got to do with vterm in particular.

[purcell]

@zeorin - unsure why you've seen that behaviour, sorry. It sounds unrelated so if it's still a problem for you, perhaps open another issue for discussion.

[purcell]

Closing this as "not planned" because I intend to stick with the simpler and more predictable existing code now that interrupting with C-g behaves well.

[psionic-k]

I didn't see a way to use change-major-mode-hook, but change-major-mode-after-body-hook hook looks viable. Quick POC:

(define-derived-mode foo-mode fundamental-mode "Foo"
  "Major mode for doing nothing.")

(define-minor-mode bar-mode
  "Minor mode to enable bar features."
  :lighter " bar"
  :global nil
  (if bar-mode
      (message "Bar mode enabled.")
    (message "Bar mode disabled.")))

(add-hook 'foo-mode-hook #'bar-mode)

(defvar foo-ready nil)

(add-hook 'change-major-mode-after-body-hook
          (lambda ()
            (when (eq major-mode 'foo-mode)
              (unless foo-ready (major-mode-suspend)))))

(defun foo-enable-and-complete-switch ()
  "Disarm the hook and load the suspended-mode"
  (interactive)
  (setq foo-ready t)
  (foo-mode))

After evaluating:

1. Open buffer
2. Switch to foo mode
3. Observe fundamental mode remains active and no bar-mode was run
4. M-x foo-enable-and-complete-switch
5. Observe foo mode is active and bar mode has been run

All that would need to happen is intercepting the mode switch and, if a direnv is detected, suspend the mode unless the cached state is fine and we can continue loading synchronously. After direnv finishes, re-use the normal (major-mode-restore) and set some sentinel value to avoid re-running direnv via the after-change-major-mode-hook

Not sure how stable major-mode--suspended has been.

I think change-major-mode-hook would be preferable, first if it wasn't buffer local, and second if we could just figure out how to get the upcoming major mode. Both major-mode and major-mode--suspended are nil in that part of the mode lifecycle. change-major-mode-after-body-hook looks like the only correct hook, but it will mean we have to run the major mode body twice when diren loads asynchronously.

[hraban]

Hi all, I wrote a PR to add async to wbolster's direnv: wbolster/emacs-direnv#82. I didn't know there were competing emacs direnv plugins and I don't know what their differences are, but please feel free to use / adapt the code. I'm happy to release it under the GPL obviously--the other project is BSD 3-clause but the PR hasn't been merged yet. FWIW I've been using it since I wrote that PR and it's been seamless so far. Implementation note: I create a temp buffer to capture the output of the separate direnv process, with a deterministic name, which automatically acts as a lock to avoid concurrent direnv calls (something you want to avoid). Any concurrent direnv calls automatically enter a (non-blocking but noisy) 1-sec sleep retry loop until they are allowed to run.

I see that you are not planning to support this first class on this package, @purcell, but I thought I'd share here anyway in case other people are interested. I don't know if the APIs of the two packages are compatible but feel free to have a look, and let me know.

I would like to keep pursuing async direnv because I use a lot of direnv + nix, and Emacs is my primary IDE, meaning I regularly end up redownloading a project's latest shell context from within Emacs.

Curious to hear others' thoughts & experiences!

[ParetoOptimalDev]

@hraban I think one difference between emacs-direnv and envrc-mode is that it avoids using global variables and instead uses buffer local variables... but it's been so long since I switched I can't remember.

[matthewbauer]

I've updated Mic92's envrc fork, pulling in latest changes and fixing some issues as well. Feel free to use it if you need this functionality:

https://github.com/matthewbauer/envrc/tree/async

[hraban]

And since we're still on the topic: @czan posted some great feedback and analysis of my PR in https://github.com/wbolster/emacs-direnv/pull/82 and long story short it's not looking like my approach was a good one. I will be trying out some of y'all's ideas here.

[purcell]

Similar opinions expressed in that PR review to mine. I still feel like the simple, predictable approach of loading synchronously but allowing a C-g escape hatch is the best one.

It occurred to me that another fairly simple approach would be to allow users to configure an execution time limit, and then automatically abort any long direnv invocation, with a warning displayed — but only while loading envrc-mode initially.

Thereafter, you could manually recover by using envrc-reload at your convenience, which would block as long as necessary, or even provide a new envrc-reload-async command for that purpose — it's not like I want everything to block, I just wouldn't try to do the async load every time, due to unpredictability of when mode hooks will run relative to the env update.

[tylerjl]

Chiming in here that I'm a happy user of @matthewbauer's async fork but I recently ran into the need for TRAMP support and so have to vacillate a bit between upstream and the async fork (and I'm happy to find TRAMP support here in the upstream repo!). It'd be lovely to merge these disparate streams together :sweat_smile: (I understand that async + TRAMP might be an entirely different question, but I'm mostly talking about the philosophical direction here regarding whether async should be supported at all)

or even provide a new envrc-reload-async command for that purpose

Given this sentiment, is there any interest in supporting async operation in an opt-in way (or, as you noted, a separate experimental command)? I'd wager that others (like myself) might be willing to toggle on a variable or use async instead of default synchronous functions to help harden and mature the feature in upstream if it were available (and accept it might be buggy until it's had time to mature). I'm not trying to volunteer anyone for that effort but did want to float the idea to gauge whether it'd be an acceptable type of pull request.

[purcell]

I'm not philosophically opposed to having opt-in support baked in for async calls, but I'm very wary of the package turning into an unmaintainable mess — it's already fairly complex IMO. I do continue to think that slow direnv executions are a direnv configuration problem, not something that should be pushed into Emacs to deal with. However, if we consider them an exception, then we could consider an alternate approach – keep the existing scheme, broadly, but allow the user to specify a timeout for direnv executions, and skip if the timeout is exceeded, at which point the user could perform an async update (manually or automatically).

[tylerjl]

@purcell I appreciate the feedback. I probably lack the elisp expertise but if I ever find some spare time to do it, I think "configurable timeout; optional async failover" seems reasonable and a modest step toward introducing some async-like capabilities.

I can totally understand the desire to avoid unnecessary complexity here. I've benefitted tremendously both from this package and the async fork (it works well to delay eglot until the environment is sourced) so hopefully there's a nice way to make it even better.

[psionic-k]

There is a clean solution available, but only by going on Emacs devel with a patch.

Emacs hooks were not really designed to support waiting or asynchronous work. A hook cannot both update the environment asynchronously and avoid returning.

What we are missing is the ability to do something asynchronous and tell Emacs to suspend further hooks for now. We also need a memento to resume hooks later. The memento can be made in the hook or made by Emacs and put into a callback for the asynchronous action, but there has to be some means of continuation.

If asynchronous support is suggested, a likely tangent is to express dependency in hooks. That has long been avoided because it requires packages to cooperate in their development.

What I wrote earlier is just a hack that runs the major mode twice if necessary. The sudden switch to fundamental would provide accidental user feedback that direnv is running. Rather than a continuation or memento, this style of solution minimizes the amount of work that is duplicated, just doing the best with what is available. I don't think this is right. It does demonstrate the problem and why a change in Emacs is necessary.

After all modes are loaded, interactive commands should be able to run asynchronously as long as there is user feedback. This conversation doesn't really have anything to do with that. If the user does something while direnv is running, they know they are taking a chance to get the old or new environment.