search

jimeh / build-emacs-for-macos

Somewhat hacky script to automate building of Emac.app on macOS.
484 stars 50 forks source link

cconv.el.gz fails to compile due to missing libgcc_ext.10.5 #5

Closed jimeh closed 4 years ago

jimeh commented 4 years ago

Currently cconv.el.gz that's part of Emacs fails to natively compile for me, leading to a error/warning at start up. It also leaves a zero byte sized *.eln file behind, which can be dealt by with the work around in #3.

As far as I can tell this doesn't seem to actually break anything or prevent things from working correctly though.

Details

The compile log error:

Compiling /Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/cconv.el.gz...
uncompressing cconv.el.gz...
uncompressing cconv.el.gz...done
ld: library not found for -lgcc_ext.10.5
libgccjit.so: error: error invoking gcc driver

Initially I thought it was simply missing the libgcc_ext.10.5.dylib from GCC, as other libs are bundled into Emacs.app, but libgcc_ext was not. However manually adding it to the libs to embed/bundle yields this error:

fatal error: /Library/Developer/CommandLineTools/usr/bin/install_name_tool: input file: /Users/jimeh/Projects/build-emacs-for-macos/sources/emacs-mirror-emacs-c818c29/nextstep/Emacs.app/Contents/MacOS/lib-x86_64-10.15/libgcc_ext.10.5.dylib is Mach-O dynamic shared library stub file and can't be changed

Sadly some googling for information about this kind of error and possible workarounds didn't yield much.

So I tried not bundling libgccjit into Emacs.app, and instead referencing it from the homebrew install path, but that didn't solve the issue either.

For now I'm out of ideas for how to fix or workaround this.

Update: It seems that cconv.el.gz compiles just fine on my machine if I launch Emacs into terminal mode:

/Applications/Emacs.app/Contents/MacOS/Emacs -nw

I'll try and dig into why it fails when launched with GUI, but doesn't in terminal.

jimeh commented 4 years ago

At some point today cconv.el.gz was natively compiled for me, cause I now have a *.eln file for it. I'm still running the exact same build of Emacs as yesterday, so I'm not sure what's going on 🤔

jakejx commented 4 years ago

I am also experiencing the same error but for subr.el. This causes failed compilation for all other packages. I'm still looking into what the cause is.

jimeh commented 4 years ago

@jakejx if it helps, cconv.el.gz has been constantly failing for me again since I switched to a newer build after that one time it did compile. It must have been some weird fluke that it didn't fail the one time ¯\_(ツ)_/¯

jakejx commented 4 years ago

hmm not sure if it matters, but I am seeing it for a different file and its causing subsequent errors:

Compiling /Users/junxuan/Downloads/build-emacs-for-macos/builds/Emacs.app/Contents/Resources/lisp/emacs-lisp/subr-x.el.gz... uncompressing subr-x.el.gz... uncompressing subr-x.el.gz...done ld: library not found for -lgcc_ext.10.5 libgccjit.so: error: error invoking gcc driver Native elisp load failed: "/Users/junxuan/.emacs.d/eln-cache/x86_64-apple-darwin19.4.0-ef3bf8ba088c7b38/subr-x-7ac74ad07543ba0e24902ea7dd76ea64-e56611d5b77b59b8916db00f6e48777a.eln", "dlopen(/Users/junxuan/.emacs.d/eln-cache/x86_64-apple-darwin19.4.0-ef3bf8ba088c7b38/subr-x-7ac74ad07543ba0e24902ea7dd76ea64-e56611d5b77b59b8916db00f6e48777a.eln, 1): no suitable image found. Did find: /Users/junxuan/.emacs.d/eln-cache/x86_64-apple-darwin19.4.0-ef3bf8ba088c7b38/subr-x-7ac74ad07543ba0e24902ea7dd76ea64-e56611d5b77b59b8916db00f6e48777a.eln: file too short /Users/junxuan/.emacs.d/eln-cache/x86_64-apple-darwin19.4.0-ef3bf8ba088c7b38/subr-x-7ac74ad07543ba0e24902ea7dd76ea64-e56611d5b77b59b8916db00f6e48777a.eln: file too short"

And I get multiple of the Native elisp load failed for all the libraries it tries to natively compile.

jimeh commented 4 years ago

@jakejx those errors actually look related to #3, and to some degree #4 as well, cause it's trying to load *.eln files from within Emacs.app itself, and my config workaround on #3 won't be removing empty *.eln files from inside of Emacs.app, just the user-specific eln-cache folder.

That said, I've not actually tried to run the Emacs.app from the builds folder myself, which means due to #4 I've not actually tried starting it with the *.eln files compiled at build time, instead it's always had to re-compile all things, including Lisp that's part of Emacs.

I'd be curious to see what happens if you copy Emacs.app to a different path, so the cached *.eln files bundled into the app are bypassed and everything is native compiled from scratch.

bramadams commented 4 years ago

Hi,

I've had the same issue regarding libgcc_ext.10.5, which seems to be solved by putting the following in your early init.el (see https://gist.github.com/mikroskeem/0a5c909c1880408adf732ceba6d3f9ab#gistcomment-3390480):

(setenv "LIBRARY_PATH" "/usr/local/opt/gcc/lib/gcc/10")

However, now emacs complains about a similar issue for "-lgcc":

ld: library not found for -lgcc

I don't find libgcc in the compiled GCC, only libgcc_s. Any ideas about this?

jakejx commented 4 years ago

@jimeh I think Emacs is trying to recompile everything from scratch, but its not able to find libgccjit.so. Which is odd because it could find it during the initial compilation.

jimeh commented 4 years ago

I seem to have find out how cconv.el was native compiled on my machine that one time. If I launch Emacs in a terminal (/Applications/Emacs.app/Contents/MacOS/Emacs -nw) it compiles it, while if I launch it as GUI app, it fails. I haven't gotten to the bottom of why yet, but I'll try and dig a bit deeper when I have time.

jimeh commented 4 years ago

@jakejx Yep, as detailed in #4, it generally need to re-compile all elisp code if the Emacs.app is moved, as the eln-cache files have a hash based on the absolute path of the lisp file being compiled.

jimeh commented 4 years ago

@bramadams I'm not sure that actually fixes the issue, looks more like it just breaks things even worse. I mentioned in my earlier comment though, it cconv.el compiles for me if I launch emacs in terminal mode, so I'd suggest trying that :)

bramadams commented 4 years ago

@jimeh Yes, I just tried, and the terminal version compiles natively. Reloading the GUI app seems to use the compiled files from the cache, but since compilation does not work at all for me in the GUI app ("library not found for -lgcc_ext.10.5" ), there are still a lot of "no suitable image found" messages. In any case, there's progress.

jimeh commented 4 years ago

Strange, in GUI when I get that error, it only prevents cconv.el.gz from compiling, everything else native compiles just fine.

I have even repeatedly reproduce the behavior by manually trashing the cconv eln-cache file, and some others, which causes the GUI to fail compiling cconv, but the rest compile again without issue.

For reference, I'm on latest macOS (10.15.6) and Xcode (11.6).

bramadams commented 4 years ago

OK, thanks, I'm on 10.13.6 and Xcode 10.1. I can't upgrade OSX, but I'll try upgrading Xcode.

bramadams commented 4 years ago

... and by the way, thanks for making these scripts available, never got this far with native compilation :-)

jakejx commented 4 years ago

I can confirm that performing the initial compilation using the terminal version has resolved the issue for me. Any compilation that arises when using the GUI after that seems to work fine. Thanks for the scripts @jimeh!

bramadams commented 4 years ago

Small update: the GUI app has native compilation fully functional when I invoke it from the command line. Only when opening the app via Finder, the libgcc_ext problem persists. It must be some kind of environment issue with app bundles.

jakejx commented 4 years ago

Hi,

I've had the same issue regarding libgcc_ext.10.5, which seems to be solved by putting the following in your early init.el (see https://gist.github.com/mikroskeem/0a5c909c1880408adf732ceba6d3f9ab#gistcomment-3390480):

(setenv "LIBRARY_PATH" "/usr/local/opt/gcc/lib/gcc/10")

However, now emacs complains about a similar issue for "-lgcc":

ld: library not found for -lgcc

I don't find libgcc in the compiled GCC, only libgcc_s. Any ideas about this?

Just an update on this. By adding one more path to LIBRARY_PATH I am able to resolve the issues. Based on what I understand from https://gcc.gnu.org/onlinedocs/jit/internals/index.html#environment-variables the path to libgcc subdirectory also needs to be set. On my system, using (setenv "LIBRARY_PATH" "/usr/local/Cellar/gcc/10.2.0/lib/gcc/10:/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin19/10.2.0") seems to do the trick.

I am not sure if there is anyway for us to specify these during the build process. Alternatively, is there some emacs variable that we can use to get the path of Emacs itself, so users can set these environment variables relative to the app bundle instead of the system gcc.

jimeh commented 4 years ago

@jakejx Thats some awesome digging, thanks :D

I'll give it a try myself, and also experiment with moving those folders into the Emacs.app itself, if that works I can have the built script do it automatically, then it's just a matter of figuring out if the relevant (setenv ...) can be baked into the the app too.

Worst case a work around could be to require users to update their config with it. Some config tweaks are already needed anyway.

jimeh commented 4 years ago

I've managed to hand-craft a Emacs.app which is fully self-contained with gcc/libgccjit, and sets the LIBRARY_PATH environment variable correctly without any user config needed. cconv.el.gz compiles just fine when launching it by double clicking in Finder.

I'll update the build script accordingly tomorrow, and hopefully it should resolve both #5 and #7 :)

jimeh commented 4 years ago

@jakejx @bramadams I've just created Pull Request #10 that updates the build script to embed GCC into Emacs.app along with also setting LIBRARY_PATH correctly on startup without any user config changes through the use of a custom launcher script.

If you wouldn't mind trying it out, that'd be great :)

Everything seems to work for me, even after I move /usr/local/Cellar/gcc/10.2.0 out of the way. I did also try it on an older mac I have which also runs macOS 10.15.6 like my main machine. But it failed to start due to weird unsupported architecture flag in the executable. It's CPU is a 2 or 3 generations older than then CPU in my current machine. So seems I'd need to test by compiling Emacs on the machine, but that kinda defeats the purposes to ensure the app is fully self-contained.

jimeh commented 4 years ago

This should now be fixed thanks to #10.

goofansu commented 3 years ago

Hi, thank you for the script.

I can successfully build out the app with ./build-emacs-for-macos -j2 feature/native-comp. But it reports faild to compile in the Warnings buffer:

Warning (comp): libgccjit.so: error: error invoking gcc driver Disable showing Disable logging
Warning (comp): Internal native compiler error: "/Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/cl-lib.el.gz", "failed to compile", "/Users/james/.emacs.d/eln-cache/28.0.50-x86_64-apple-darwin19.6.0-deda61b35cbed626661f68bc7cd413a9/cl-lib-8b9389005ae118dfb032ca52ea69e8d9-28984542edce56cce7e53ec8ccd624af.eln", "error invoking gcc driver" Disable showing Disable logging
Warning (comp): seq.el.gz:395:16: Warning: ‘seq-contains’ is an obsolete generic function (as Disable showing Disable logging
Warning (comp): libgccjit.so: error: error invoking gcc driver Disable showing Disable logging
Warning (comp): Internal native compiler error: "/Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/seq.el.gz", "failed to compile", "/Users/james/.emacs.d/eln-cache/28.0.50-x86_64-apple-darwin19.6.0-deda61b35cbed626661f68bc7cd413a9/seq-c314b42f12831596cfc86f40bcdbfc80-5a24074f219b51ac0086349bdb7f5622.eln", "error invoking gcc driver" Disable showing Disable logging
Warning (comp): libgccjit.so: error: error invoking gcc driver Disable showing Disable logging
Warning (comp): Internal native compiler error: "/Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/gv.el.gz", "failed to compile", "/Users/james/.emacs.d/eln-cache/28.0.50-x86_64-apple-darwin19.6.0-deda61b35cbed626661f68bc7cd413a9/gv-e0cf7478c665ac9dd903b899257854d6-07285e7deb1fdfdad07c352586e0d756.eln", "error invoking gcc driver" Disable showing Disable logging
Warning (comp): libgccjit.so: error: error invoking gcc driver Disable showing Disable logging
Warning (comp): Internal native compiler error: "/Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/cl-extra.el.gz", "failed to compile", "/Users/james/.emacs.d/eln-cache/28.0.50-x86_64-apple-darwin19.6.0-deda61b35cbed626661f68bc7cd413a9/cl-extra-d7051cba75a4863ffba77470a08a226d-d670409d2537bb7bf4fee93d0922c469.eln", "error invoking gcc driver" Disable showing Disable logging
Warning (comp): libgccjit.so: error: error invoking gcc driver Disable showing Disable logging
Warning (comp): Internal native compiler error: "/Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/easymenu.el.gz", "failed to compile", "/Users/james/.emacs.d/eln-cache/28.0.50-x86_64-apple-darwin19.6.0-deda61b35cbed626661f68bc7cd413a9/easymenu-1fb41c0e78f9aa8e9de83e9219437950-c3486f53c54967059ad37854fd3845bd.eln", "error invoking gcc driver" Disable showing Disable logging
Warning (comp): libgccjit.so: error: error invoking gcc driver Disable showing Disable logging
Warning (comp): Internal native compiler error: "/Applications/Emacs.app/Contents/Resources/lisp/help-mode.el.gz", "failed to compile", "/Users/james/.emacs.d/eln-cache/28.0.50-x86_64-apple-darwin19.6.0-deda61b35cbed626661f68bc7cd413a9/help-mode-d4dbae3d39e6a79cefe8c3ec980fa59d-13fbdfb7fa0b4403fd118315b412e149.eln", "error invoking gcc driver" Disable showing Disable logging
...
...

build-emacs-for-macos: 9223ff8e macOS: 10.15.7

Update

~/.emacs.d (develop|✚3) [255] $ doom install
ld: library not found for -lgcc_ext.10.5
libgccjit.so: error: error invoking gcc driver

Update 2: 

~/.emacs.d (develop|✚3) [255] $ env LIBRARY_PATH=/usr/local/opt/gcc/lib/gcc/10/ doom install
ld: library not found for -lgcc
libgccjit.so: error: error invoking gcc driver
ld: library not found for -lgcc
libgccjit.so: error: error invoking gcc driver
No catch for tag: exit, 255

Problem resolved: Thanks to https://github.com/jimeh/build-emacs-for-macos/issues/5#issuecomment-683426475, I set LIBRARY_PATH to /usr/local/Cellar/gcc/10.2.0/lib/gcc/10:/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin18/10.2.0, and it works.

meain commented 3 years ago

For some reason setting library path as the one in /usr/local/Cellar did not seem to help me. But I could see that there was lib directory in the Emacs.app package and setting that as LIBRARY_PATH seemed to help. If you have Emacs.app in your /Applications folder add this to the early-init.el file.

(setenv "LIBRARY_PATH" "/Applications/Emacs.app/Contents/MacOS/lib/gcc/10:/Applications/Emacs.app/Contents/MacOS/lib/gcc/10/gcc/x86_64-apple-darwin17/10.2.0")

It looks like patches/native-comp-env-setup.patch) is doing this but for some reason it does not seem to work.