apple m3 universal binaries incomplete

[informatimago]

In the binary downloaded from http://emacsformacosx.com

Emacs Version 29.2 Universal Binary (95.907 MB) Released 2024-01-18

we find tools compiled only for x86_64:

$ file /Applications/Emacs.app/Contents/MacOS/{Emacs,libexec/*}
/Applications/Emacs.app/Contents/MacOS/Emacs:            Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64]
/Applications/Emacs.app/Contents/MacOS/Emacs (for architecture x86_64): Mach-O 64-bit executable x86_64
/Applications/Emacs.app/Contents/MacOS/Emacs (for architecture arm64):  Mach-O 64-bit executable arm64
/Applications/Emacs.app/Contents/MacOS/libexec/hexl:     Mach-O 64-bit executable x86_64
/Applications/Emacs.app/Contents/MacOS/libexec/movemail: Mach-O 64-bit executable x86_64
/Applications/Emacs.app/Contents/MacOS/libexec/rcs2log:  POSIX shell script executable (binary data)

so they don't work eg. on apple m3 arm64.

[informatimago]

Actually, these tools are compiled separately for different architectures, and libexec is a symlinl:

/Applications/Emacs.app/Contents/MacOS/libexec -> libexec-x86_64-10_11/

but it points to the bad directory on arm.

Fat executables could be built with the lipo tool.

lipo -create arm64-test x86_64-test -output universal-test

[caldwell]

Inside Emacs, do M-x describe-variable RET exec-path (or C-h v exec-path RET). It should have the correct directory listed first. On my M1 I have these as my last 3 entries:

"/Applications/Emacs.app/Contents/MacOS/bin-arm64-11"
"/Applications/Emacs.app/Contents/MacOS/libexec-arm64-11"
"/Applications/Emacs.app/Contents/MacOS/libexec"

I could lipo the arm and one of the x86_64 arches together, but you'll notice that it ships with 2 x86_64 arches (one for very old OSes). Unfortunately lipo (and Mach-O in general, I believe) only handles one image per arch, leading to the current "why bother with lipo" approach.

The symlinked archless libexec dir was really just for usage outside of emacs—I used to use hexl a lot in the terminal and wanted a consistent place to symlink it into my PATH. I don't remember the reason that I included it in the exec-path, but it shouldn't cause problems since it is after the correct arch which has all the same-named binaries in it. I am able to use M-x hexl-mode, for instance.

What problem is this causing you, specifically?

[informatimago]

It was chatgpt-shell that was failing on it, when it started a process "hexl". I have a exec-path with a lot of additionnal directories, and ineed, with libexec-arm64-11, so it should have worked. So it must be the way chatgpt-shell launches hexl that is bad? git@github.com:xenodium/chatgpt-shell.git I understand the problem with the different OS versions. I just updated the symlinks instead.

[caldwell]

If you open your terminal and type which hexl does it print anything (that is, do you have hexl somewhere if your normal shell PATH)?

I took a quick peek at the chatgpt-shell and it looks like this might be the place where it calls hexl. I'm not sure why except maybe it's using it as a sanity test to see if executing things works? It doesn't look like it's doing anything obvious that would cause it to run the wrong hexl…

[informatimago]

If you open your terminal and type which hexl does it print anything (that is, do you have hexl somewhere if your normal shell PATH)?

I took a quick peek at the chatgpt-shell and it looks like this might be the place where it calls hexl. I'm not sure why except maybe it's using it as a sanity test to see if executing things works? It doesn't look like it's doing anything obvious that would cause it to run the wrong hexl…

Yes, I have a rather full PATH:

$ which hexl
/Applications/Emacs.app/Contents/MacOS/libexec/hexl

The help of start-process mentions exec-path: PROGRAM is the program file name. It is searched for in ‘exec-path’ In my .emacs, I prefix the contents of PATH and others to my exec-path.

exec-path -->
("/opt/local/bin" "/opt/local/sbin" "/usr/local/bin" "/Users/pjb/bin/" "/bin" "/Users/pjb/bin" "/opt/haskell-language-server/bin" "/opt/local/libexec/gnubin" "/opt/local/sbin" "/opt/local/bin" "/opt/local/libexec/rbenv" "/opt/X11/bin" "/usr/local/bin" "/usr/X11R6/bin" "/usr/X11/bin" "/usr/sbin" "/usr/bin" "/sbin" "/Users/pjb/bin/" "/Applications/Emacs.app/Contents/MacOS/bin-arm64-11_2" "/Applications/Emacs.app/Contents/MacOS/libexec-arm64-11_2" "/Applications/Emacs.app/Contents/MacOS/libexec" "/Applications/Emacs.app/Contents/MacOS/bin" "/Applications/Emacs.app/Contents/MacOS/bin-arm64-11" "/Applications/Emacs.app/Contents/MacOS/libexec-arm64-11" "/Applications/Emacs.app/Contents/MacOS/libexec")

[caldwell]

Here are the emacs portions of your exec-path:

"/Applications/Emacs.app/Contents/MacOS/bin-arm64-11_2"
"/Applications/Emacs.app/Contents/MacOS/libexec-arm64-11_2"
"/Applications/Emacs.app/Contents/MacOS/libexec"
"/Applications/Emacs.app/Contents/MacOS/bin"
"/Applications/Emacs.app/Contents/MacOS/bin-arm64-11"
"/Applications/Emacs.app/Contents/MacOS/libexec-arm64-11"
"/Applications/Emacs.app/Contents/MacOS/libexec")

The last 3 are the ones added by the launcher. The rest I'm guessing are coming from your shell stuff. In particular bin-arm64-11_2 looks like it was copied from an older emacs before I trimmed the "new-style-greater-than-10" versions back to just the major version number. Since those dirs don't exist any more, the first valid path is plain libexec and so it's finding the x86_64 binary.

The solution would be to change those paths in your .bashrc or .zsh-whatever (can't remember what that one's called). I'd consider removing the plain libexec one.

I have been thinking about this since yesterday and it's annoying to have moving path targets inside the emacs (those OS version numbers at the end could change at any moment). It would be nice to either have an architecture symlink with no version numbers in it or just lipo up the plain directory as you originally suggested. That's probably the least surprising thing from a user perspective.

[informatimago]

Thanks, I'll check my paths.

Also, since we can install it by just dragging Emacs.app somwhere. And it could be installed on a network file system and used simultaneously by intel or arm systems. So updating the symlinks at whatever time wouldn't be a good solution indeed.