Native trampoline bootstrap issues / missing gcc driver?

[shipmints]

On macos monterey 12.7, I've been trying to debug this and work around it but I think there's something missing from the gcc in the Emacs build.

If I launch your Emacs build from the command line, the subr---trampoline files are correctly created. If I delete the eln-cache and start Emacs from Spotlight (Command-Space app launching), Emacs refuses to bootstrap. I can get around this by launching once via command line and thereafter from Spotlight.

I see you've cleverly embedded the gcc / libgccjit distribution so it should work.

I get this error in Emacs Messages

x86_64-apple-darwin21-gcc-14: fatal error: cannot execute 'as': posix_spawnp: No such file or directory

You should be able to reproduce this on the command line via

env -i /Applications/Emacs.app/Contents/MacOS/Emacs

It works correctly when I start with homebrew in my path

env -i PATH=/usr/bin:/usr/local/bin /Applications/Emacs.app/Contents/MacOS/Emacs

I can see x86_64-apple-darwin21-gcc-14 in my homebrew path, so I guess it's that command that's missing from the embedded gcc build? Your Emacs build seems otherwise self contained.

[jimeh]

I'm on mobile, so I can't verify this right now, but if memory serves me right, the missing as binary is part of Xcode or Xcode Command Line Tools, not gcc.

Possibly there's some initial env setup issue via the app bundle plist file so the relevant path to make as available is missing.

I'll update with more details when I've had a chance dig into it some more.

[shipmints]

Thanks for taking a look. I'm not sure it's just the plist as I just checked the embedded gcc distro and it doesn't have any command-line drivers.

[jimeh]

Sorry I haven't gotten back around to this sooner, been both busy and under the weather for a few weeks.

I've had a deeper dig into this, and while your env -i [...] does indeed yield that result, it does so by blanking out all environment paths, meaning Emacs can't find any external programs at all, not even ls needed by dired.

What should happen, if you launch Emacs normally via Finder or Spotlight, is that it has default PATH / exec-path values, which includes where the as binary from Xcode is located.

I tested with a fresh macOS (14.4) VM with only Xcode installed, and could not reproduce your issue :(

Installing the latest stable 29.4 build, and launching Emacs via Finder or Spotlight, I get the following results using eval-expression (M-:):

• (getenv "PATH"):

  "/usr/bin:/bin:usr/sbin:/sbin"

• exec-path:

  ("/usr/bin" "/bin" "/usr/sbin" "/sbin" "/Applications/Emacs.app/Contents/MacOS/libexec")

• (executable-find "as"):

  "/usr/bin/as"

On this VM, the as binary which the embedded GCC calls is located at /usr/bin/as. I tested with a empty init.el containing only Melpa setup, and ran:

M-x package-install RET doom-modeline RET

This resulted in a bunch of natively compiled *.eln files:

Screenshot


So, my theory for your issue, is that for some reason, the default PATH that Emacs ends up with when started via Finder or Spotlight, does not include /usr/bin. So I'd be curious to see what exec-path and PATH is within your Emacs instance launched via Finder/Spotlight. Ideally with a completely empty init.el.

I did also have a quick attempt at modifying Emacs.app/Contents/Info.plist to add a LSEnvironment value with a custom PATH env var, but it did not work. So I'm not sure if an app can force specific env vars to be added anymore. But worst case I might be able to work around it with another check and tweak in the custom site-start.el which I add to setup GCC and native comp, essentially setting up defaults if PATH is empty on start up.

[shipmints]

Thanks for the extra effort. I just verified that my paths are the same as the ones you reported after starting Emacs from Spotlight and after moving my early-init.el and init.el files out of the way. It's something else besides that, I suspect.

My gcc is the following, in case that's a useful difference.

$ gcc --version
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: x86_64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

[shipmints]

I also verified that the eln-cache is populated in ~/.emacs.d with trampolines. Perhaps it's due to that in my early-init.el I relocate eln-cache as follows and that somehow screws with Emacs bootstrap and this might bite other people:

(defconst my:emacs-var-root (expand-file-name "var" user-emacs-directory))
(defconst my:arch (car (split-string system-configuration "-"))) ; e.g., "x86_64" "aarch64"
(when (native-comp-available-p)
  (require 'comp)
  (setq native-comp-async-report-warnings-errors 'silent)
  (setq-default native-comp-async-jobs-number 0) ; 0 means use 1/2 available CPUS
  (setq native-comp-enable-subr-trampolines t)
  (setq native-comp-jit-compilation t)
  (startup-redirect-eln-cache
   (expand-file-name (concat (file-name-as-directory "eln-cache")
                             my:arch) my:emacs-var-root))
)

I'll try once more with a tiny early-init.el and a basically empty init.el.

[shipmints]

I started Emacs as before with that relocated eln-cache and in *scratch* just did (require 'project) and that got compiled BUT there are no trampolines. So there's something going on with a relocated eln-cache and this build.

[shipmints]

Question for you. Is it expected that users of this build have xcode installed? i.e., the build isn't expected to be self contained including the gcc drivers needed?

[jimeh]

Yeah, Xcode Command Line Tools or Xcode proper are a requirement for using native compilation.

As far as I'm aware that's because GCC on macOS needs to call out to Apple's assembler (the as command), and the assembler is part of Xcode. So builds are as self-contained as can be given the platform constraints. Though if you don't use native compilation, then Xcode stuff isn't needed.

In theory I guess it might be possible to embed Xcode Command Line Tools into the app bundle, but that's another like 200MB or something if it's doable at all, and it's likely to be exceptionally horrible and hacky. So I'd be very reluctant to go down that road.

Though, on systems that don't have Xcode stuff installed, the first time native compilation runs, macOS pops a dialog prompting you install Xcode Command Line Tools.

[shipmints]

Makes sense and you actually did say it in the README... Still not sure what's preventing trampolines. Let's keep this issue open until we figure it out. Clearly not a giant showstopper but it is one annoyance for new users that may encounter needing to first launch via command line and then via Spotlight/Finder from thereon.

[jimeh]

The version report you're getting for gcc is actually from Apple's clang compiler that exposes a gcc compatible mode.

That shouldn't really matter however, as the native compilation stuff should be using the embedded GCC and libgccjit libraries.

Did you try (executable-find "as")?

Also, you can try (getenv "LIBRARY_PATH"), it should contain paths pointing at the embedded gcc within the app bundle.

[jimeh]

Sure, I'm happy to keep the issue for now in case anyone else runs into it.

Only big difference between your setup and the fresh VM I tested on, is that you're on macOS 12.x, while my VM is 14.x.

When I next have some time, I'll see if I can get a macOS 12.x VM running, the 14.x one I used tonight was still laying around from a couple of months ago. But it had a snapshot for right after the OS was initially installed, so that made it quick and easy to get going :)

[shipmints]

Yes, "as", for me, is in the precise location you reported, albeit the clang wrapper but that's what it will always be via xcode.

[shipmints]

(getenv "LIBRARY_PATH") is nil for me.

[jimeh]

Ah, LIBRARY_PATH being nil would definitely break native compilation.

It should be something like this:

"/Applications/Emacs.app/Contents/Frameworks/gcc/14:/Applications/Emacs.app/Contents/Frameworks/gcc/14/gcc/aarch64-apple-darwin22/14:/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib"

It should be setup via Emacs.app/Contents/Resources/lisp/site-start.el within the app bundle, which contains something similar to this, but customized for the specific version of GCC that's embedded in the app bundle:

;; Allow Emacs to find bundled C sources.
(setq source-directory
  (expand-file-name ".." (file-name-directory load-file-name)))

;; Set LIBRARY_PATH to point at bundled GCC and Xcode Command Line Tools to
;; ensure native-comp works.
(when (and (eq system-type 'darwin)
           (string-match-p "\.app\/Contents\/MacOS\/?$"
                           invocation-directory))
  (let* ((library-path-env (getenv "LIBRARY_PATH"))
         (devtools-dir
          "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib")
         (gcc-dir (expand-file-name
                   "../Frameworks/gcc/14"
                   invocation-directory))
         (darwin-dir (expand-file-name
                      "../Frameworks/gcc/14/gcc/aarch64-apple-darwin22/14"
                      invocation-directory))
         (lib-paths (list)))

    (if library-path-env
        (push library-path-env lib-paths))
    (if (file-directory-p devtools-dir)
        (push devtools-dir lib-paths))
    (push darwin-dir lib-paths)
    (push gcc-dir lib-paths)

    (setenv "LIBRARY_PATH" (mapconcat 'identity lib-paths ":"))))

The site-start.el file should be automatically executed by Emacs on start, unless you've passed in a --no-site-file, -Q or other flag which skips it.

You could potentially verify what invocation-directory is set to, and if it matches the pattern expected by the site-start.el file by running:

(string-match-p "\.app\/Contents\/MacOS\/?$" invocation-directory)

[shipmints]

And there we go. Turns out that I've been carrying something in my startup files for, um, AGES, that disabled site-run. I still wonder how it worked at all to bootstrap even without that from the command line, or at all, unless the ambient xcode tooling suffices along with the plist settings?

[jimeh]

Ah, nice. It's possible native comp when working for you, actually found gcc and libgccjit via homebrew if you've got both installed there. If not, I'm not really sure how it ever worked either... lol

That said, the site-start.el approach causes issues with doom-emacs as well, so I do plan to see if I can come up with another approach that patches the required stuff directly in as a after-pdump-load-hook callback, which would allow LIBRARY_PATH to still be set correctly when --no-site-file and similar options are used.

[shipmints]

Minor stylistic comments on site-run:

...
    (if library-path-env
        (push library-path-env lib-paths))
    (if (file-directory-p devtools-dir)
        (push devtools-dir lib-paths))
    ;; better as
    (when library-path-env
        (push library-path-env lib-paths))
    (when (file-directory-p devtools-dir)
        (push devtools-dir lib-paths))
...
(setenv "LIBRARY_PATH" (mapconcat 'identity lib-paths ":"))))
;; is better as
(setenv "LIBRARY_PATH" (mapconcat #'identity lib-paths path-separator))))

[shipmints]

FYI, I was able to bootstrap no problem from an empty relocated eln-cache now that site-run is active.