emacsWithPackages' clearing of EMACSLOADPATH breaks emacs-async

[hraban]

Describe the bug

emacs-async is an emacs package which spawns a separate Emacs process to do async work. This workflow is broken by the current implementation of emacsWithPackages, which uses the EMACSLOADPATH envvar to share the path for packages with the emacs process: it also adds an init file which clears that envvar after it has been read by Emacs. This means that once you start emacs-async, that forked Emacs won't have access to any of the plugins.

The commit that introduced that clearing of the envvar is here: "https://github.com/NixOS/nixpkgs/pull/106486/commits/23d4bfb6661ca57a9e331a2cf4184232d38ac38b" it also explains the motivation.

The nixpkgs-specific init file that clears the envvar is here: https://github.com/alyssais/nixpkgs/blob/165da28db301f89c7c9cfce7564d1b278996af56/pkgs/applications/editors/emacs/site-start.el#L17-L27

It would be nice to have an easy way to fork the current emacs, with all packages available. Whether that is by somehow having a reference to the original wrapper script, or by setting the envvars, or by not clearing them. I'm not sure which one I think is best, tbh.

Steps To Reproduce

Working example

Child doesn't need any packages:

nix shell --impure --expr '((import <nixpkgs> {}).emacsWithPackages (e: [ e.dash e.async ])).overrideAttrs (_: { pname = "emacs"; })' --command emacs --batch --eval "
(progn
  (package-initialize)
  (require 'dash)
  (-> \"dash loaded in parent\" message)
  (require 'async)
  (->> (async-start
         (lambda ()
           (message \"hello from sentinel\")))
       async-get
       (message \"parent received: %s\")))"

output:

dash loaded in parent
parent received: hello from sentinel

Broken example

Same, but try to load dash inside the client:

nix shell --impure --expr '((import <nixpkgs> {}).emacsWithPackages (e: [ e.dash e.async ])).overrideAttrs (_: { pname = "emacs"; })' --command emacs --batch --eval "
(progn
  (package-initialize)
  (require 'dash)
  (-> \"dash loaded in parent\" message)
  (require 'async)
  (->> (async-start
         (lambda ()
           (require 'dash)
           (-> \"dash loaded in sentinel\" message)))
       async-get
       (message \"parent received: %s\")))"

output:

dash loaded in parent
Debugger entered--Lisp error: (file-missing "Cannot open load file" "No such file or directory" "dash")
  async-handle-result(identity (async-signal (file-missing "Cannot open load file" "No such file or directory" "dash")) #<buffer *emacs*>)
  async-get(#<process emacs>)
  (->> (async-start (lambda nil (require 'dash) (-> "dash loaded in sentinel" message))) async-get)
  (message "parent received: %s" (->> (async-start (lambda nil (require 'dash) (-> "dash loaded in sentinel" message))) async-get))
  (->> (->> (async-start (lambda nil (require 'dash) (-> "dash loaded in sentinel" message))) async-get) (message "parent received: %s"))
  (->> (async-start (lambda nil (require 'dash) (-> "dash loaded in sentinel" message))) async-get (message "parent received: %s"))
  (progn (package-initialize) (require 'dash) (-> "dash loaded in parent" message) (require 'async) (->> (async-start (lambda nil (require 'dash) (-> "dash loaded in sentinel" message))) async-get (message "parent received: %s")))
  command-line-1(("--eval" "\n(progn\n  (package-initialize)\n  (require 'dash)\n ..."))
  command-line()
  normal-top-level()

Notify maintainers

@alyssais and @adisbladis

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-darwin"`
 - host os: `Darwin 22.5.0, macOS 13.4`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.15.1`
 - channels(user): `""`
 - channels(root): `""`
 - nixpkgs: `/nix/store/fqjg9y9pwh9cg2izs76kkv0wwwpsqp1v-source`

[hraban]

Forgot to notify the emacs maintainers: @AndersonTorres @adisbladis @Atemu @jwiegley @lovek323 @matthewbauer, as requested by the github bug report template.

[adisbladis]

I did some quick checking of the emacs-async sources and they use this to find what path to invoke:

(expand-file-name invocation-name invocation-directory)

I don't know what the right solution is to make this work as setting invocation-directory to the wrapper is bound to fail in various interesting ways and reverting the EMACSLOADPATH is not an option either because of correctness issues.

[hraban]

Thanks for your reply. What correctness issues exactly do you mean? E.g. when I think of baking PATH using wrapProgram, I would in fact want that PATH envvar to be propagated to any spawned child process.

Looking at emacs-async and the load-path some more, I've found this work-around:

nix shell --impure --expr '((import <nixpkgs> {}).emacsWithPackages (e: [ e.dash e.async ])).overrideAttrs (_: { pname = "emacs"; })' --command emacs --batch --eval "
(progn
  (package-initialize)
  (require 'dash)
  (-> \"dash loaded in parent\" message)
  (require 'async)
  (->> (async-start
         \`(lambda ()
             ,(async-inject-variables \"load-path\")
             (require 'dash)
             (-> \"dash loaded in sentinel\" message)))
       async-get
       (message \"parent received: %s\")))"

Although it does feel like a hack. Isn't emacs-async's desire reasonable, to "spawn a fresh, functional copy of this Emacs process, with access to all packages"? Or e.g. when launching a child Emacs process for another reason, wouldn't I also want access to the packages, there? Unless it is wrapped by its own wrapper script which bakes other packages, perhaps. And if I don't, I'd expect to have to clear them manually.

What do you think?