native-comp branch option

[seagle0128]

The native compiling is a amazing feature to improve the Emacs performance. I'd like get an option to build this branch. Thanks!

https://github.com/emacs-mirror/emacs/tree/feature/native-comp

[d12frosted]

@seagle0128 hey, do you use any other options? I just need to check if other patches work well together with native-comp branch.

[seagle0128]

@d12frosted I am using --with-cocoa, --with-jansson and --with-xwidgets. For example,

brew install emacs-plus --HEAD --with-jansson --without-spacemacs-icon --without-imagemagick@7 --with-cocoa --with-xwidgets

[d12frosted]

@seagle0128 thanks. I will check if it builds with these options. I can't test it locally, so has to run on CI.

[seagle0128]

Thanks a lot, @d12frosted .

[d12frosted]

Don't really understand the error:

make[2]: *** [emacs-lisp/autoload.elc] Bus error: 10
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [emacs-lisp/macroexp.elc] Bus error: 10
make[2]: *** [emacs-lisp/byte-opt.elc] Bus error: 10
make[2]: *** [emacs-lisp/cconv.elc] Bus error: 10
make[2]: *** [emacs-lisp/bytecomp.elc] Bus error: 10
make[1]: *** [bootstrap-emacs.pdmp] Error 2
make: *** [src] Error 2

Apparently, this branch requires something special to build. I will take a look once I have more time and ability to build locally.

[d12frosted]

Quick duckduckgoing leads me to https://akrl.sdf.org/gccemacs.html

Apparently, libgccjit must be installed (bundled with gcc?).

[seagle0128]

Bad news. I found this thread: https://lists.gnu.org/archive/html/emacs-devel/2019-08/msg00039.html.

[d12frosted]

@seagle0128 oh well.

[AndreaCorallo]

Hi, I know zero about Mac and related but maybe this can help?

https://gist.github.com/mikroskeem/0a5c909c1880408adf732ceba6d3f9ab

[seagle0128]

Wow, it seems a good news. Thanks, @AndreaCorallo.

[d12frosted]

@AndreaCorallo nice find! It will help.

---

I'd like to do it sooner, but starting today I will have pretty limited availability till April. If someone wants to get it ASAP, the best way to do it is to create a PR.

[akermu]

The --native-comp branch option doesn't add the --with-nativecomp option to ./configure. Otherwise I got it to compile (did not try with the formula yet), see the thread from the gist above.

[d12frosted]

I went over the instructions and updated my PR (#188). It seems that I am falling into the same trap as @AndreaCorallo . So for now it doesn't compile.

As far as I understand, this feature should bring better performance. I would love to see it 😸 . Also it feels like an interesting direction for Emacs, so I would love to support its development.

Now, when it comes to maintenance, I think it will be a burden to me.

Firstly, it requires modified version of the gcc formula. It's not a problem to provide it as well (see my PR), but it opens up a whole new world of troubles and questions. Should it shadow gcc or be a separate package (e.g. gcc-jit)? What to do with bottles in the latter case (now that it has different name, bottles can't be reused). Building from sources takes a lot of time. How to maintain versions? Trust me, I will forget to update versions, as I don't keep an eye on GCC development at all. Which means that if the user at some point of time installed gcc-jit before installing gcc he might end up with old version of gcc.

Secondly, it adds a lot of indirection and complexity to the formula. I can deal with it, though I am favouring making the formula simpler.

And most importantly, it is highly unlikely that I will be able to provide any support if it doesn't compile. So I will just have to redirect or plainly close these issues. And if it's still in active development, I suspect it will break time to time. Even HEAD breaks, right? 😸

Having all this in mind, I am not against adding this feature. The most important requirement for me is having someone willing to support it. Just let me know if you want to step in ;)

[jaidetree]

A smoother emacs experience would be delightful! Asking if it could be compiled on OS X in the Doom Discord lead me to http://akrl.sdf.org/gccemacs.html#org31d15b7. Maybe there's something new or helpful?

[CarlQLange]

Following these instructions worked for me building it locally: https://gist.github.com/AllenDang/f019593e65572a8e0aefc96058a2d23e

[d12frosted]

Oh nice. I shall give it a try! Thanks for sharing 😸

[CarlQLange]

No problem! Unfortunately it doesn't solve the gcc +jit issue. I wonder who maintains the gcc formula and if we can just PR the two flags we need... But it does sort of seem like if you have the correct gcc built, it's basically just a matter of pointing the native-comp branch at the new gcc with LIBRARY_PATH and building with --nativecomp... I, for one, would really like to use this formula again instead of having to build locally 👍

[d12frosted]

@CarlQLange yes, understandable.

I think emacs-plus should provide nativecomp branch as a separate formula (maybe emacs-plus-native) so it doesn't interfere with other formulas. Though I am fine if it will be part of emacs-plus@28 in case it will not greatly increase complexity. Which I doubt 😸

Now, one of the biggest challenges is special version of gcc. Ideally the flag should be part of the official gcc formula. I don't know specific person, but the starting point is the homebrew repository itself. In case they refuse, we can actually create a fork of gcc (like I did in the linked PR #188) and use it as dependency for native-comp formula.

So I'd say the first step is to ask brew maintainers to add jit option since it will greatly decrease complexity of implementation on our side (no need to maintain irrelevant for emacs-plus gcc formula).

[CarlQLange]

Yep, I agree. I do think it would be fine to have as emacs-plus --with-nativecomp since I think the general build process looks basically the same to me, plus or minus a flag here and a bug there.

The gcc formula is here https://github.com/Homebrew/homebrew-core/blob/master/Formula/gcc.rb. Unfortunately I don't have the time this morning to build gcc a few times to pass the PR tests (although I have the fork ready here: https://github.com/CarlQLange/homebrew-core/commit/b165dd586686cd6e47296fc78f70938b6b09cdb1), but if nobody else has done it by tomorrow, I can do it all then. I would have liked to add a flag for --with-jit but it turns out that homebrew/core does not want build.with or build.without in its formulae.

I'm not super well versed in exactly what effect building with jit by default would have, and I'm a bit wary of accidentally breaking every computer in the world by adding it ;)

[d12frosted]

I am already on PR for gcc. Takes a lot to build 😸 Regarding options - yes, they worked hard on removing most of the options from core formulas.

I'm not super well versed in exactly what effect building with jit by default would have, and I'm a bit wary of accidentally breaking every computer in the world by adding it ;)

Oh my, I thought about it as well. But I think it's better to discuss in the PR related to gcc itself.

Update: https://github.com/Homebrew/homebrew-core/pull/59134

[CarlQLange]

Ah, fantastic, thanks very much! 🤞

[d12frosted]

Well, that thread is silent 😢

[d12frosted]

I've found one interesting PR: https://github.com/Homebrew/homebrew-core/pull/34865/files#diff-baf58e557d0debf0bd219be97ec02ab8L60. So at some point of time there was an option to build with jit support, but then it was removed.

Meanwhile, I think we can actually create a gcc formula in tap and work with that. I am pretty sure that I won't be able to do it until next month, so if someone wants to dig into this - you are more than welcome 😸 You can take a look at #188 and this guide as a starting point.

[AndreaCorallo]

@d12frosted also this may help you https://github.com/shshkn/emacs.d/blob/master/docs/nativecomp.md (if you haven't seen it yet), just skip the speed 3 setting.

[jimeh]

@d12frosted in case it's of any help, I've managed to get my own build script that produces a self-contained Emacs.app working: https://github.com/jimeh/build-emacs-for-macos#native-comp

(Please excuse the messy code, it's pretty much the dirtiest hack code I have at the moment 🤦‍♂️)

It's not without some issues/annoyances though, which I've started documenting as Issues.

Despite that, everything I do on a daily basis seems to work just fine, as I've been using native-comp builds all week for work without any issues.

[seanfarley]

I've completely given up on Homebrew upstream. Is there a way we can just add libgccjit to this tap? If not, I think I might just dust off my Macports committer permissions and push there (only half joking 😰)

[CarlQLange]

Me, too. So much so that I've actually switched to Pop!_OS... Where, it must be said it is also still quite hard to get native comp up and running 😅

[d12frosted]

@seanfarley and @CarlQLange I completely understand your feelings 😸 thank you for support!

Now, since the upstream stagnates, we can actually push gcc-jit to this tap. I still have my concerns regarding support, but at least we can try and see how it goes.

FYI (just to align expectations), I have limited availability these days, but hopefully I will find time to push that in the following weeks.

[seanfarley]

Just to make sure: I really appreciate your work, @d12frosted! Your patience and skin is far thicker than mine 😄 No worries about your time; it's your attitude and support I really like. Thanks again!

[CarlQLange]

Just to make sure: I really appreciate your work, @d12frosted! Your patience and skin is far thicker than mine smile No worries about your time; it's your attitude and support I really like. Thanks again!

Hear hear!

[jsigman]

Third

[jsigman]

As I understand, everything is blocked by this: https://github.com/Homebrew/homebrew-core/pull/60338 Correct?

[d12frosted]

@jsigman no, not really. We still can follow the plan of making a gcc+jit formula in this tap, though it has several disadvantages. Some of them are described in this comment: https://github.com/d12frosted/homebrew-emacs-plus/issues/187#issuecomment-623868439

Regarding the blockers for formula in this tap - time :)

[nyamada]

Homebrew/homebrew-core#60338 looks unblocked now -- so I hope we can get this soon!

[jimeh]

@d12frosted in case it's of any help again, I've managed to get my custom build script working with the gcc and libgccjit Homebrew formulas now :)

[d12frosted]

@jimeh thank you! Glad to hear that your script works :)

There is WIP https://github.com/d12frosted/homebrew-emacs-plus/pull/188 that I am working on. It already builds, but Emacs doesn't start.

[d12frosted]

188 is merged! it is red in CI, but builds locally for me

please let me know if it doesn't work for you, or if you encounter any issues!

[DaniruKun]

@d12frosted shouldn't the README be updated also with this option? Or you will do it anyway after a bit?

[d12frosted]

@DaniruKun It will be updated a little bit later. First I want to get some confirmations that it works not only for me :) So let's say, native comp is in a closed beta 😸 only for those who read this thread 😸

[DaniruKun]

@DaniruKun It will be updated a little bit later. First I want to get some confirmations that it works not only for me :) So let's say, native comp is in a closed beta 😸 only for those who read this thread 😸

Thanks for all the work guys! Will give it a try tomorrow, and ask a colleague to have a go too, will see how it goes 🤞

[nyamada]

thanks for all the work! if it works, it will be a nice xmas present for everyone --

[julienfantin]

Despite a successful install, Emacs seems unable to natively compile libraries, failing with these messages:

Warning (comp): collect2: error: ld returned 1 exit status 
Warning (comp): libgccjit.so: error: error invoking gcc driver
Warning (comp): Debugger entered--Lisp error: (native-ice "/usr/local/Cellar/emacs-plus@28/28.0.50/share/emac..." "failed to compile" "/Users/jfantin/.emacs.d/eln-cache/28.0.50-x86_64-a..." "error invoking gcc driver")

A couple of failed attempts at resolving the isssue:

• brew doctor had a warning about libgccjit being unlinked, so I tried brew link --overwrite libgccjit
• rm -rf ~/.emacs.d/straight ~/.emacs.d/elpa ~/.emacs.d/eln-cache

[jimeh]

@julienfantin That sounds to me like an issue with the LIBRARY_PATH environment variable. I had to solve that for my self-contained builds via a hacky patch.

From what I recall it could find the homebrew installed GCC and libgccjit when running or launched from a terminal, but not when launched into GUI mode via Finder.

To test if that's the issue, try launching Emacs into terminal mode:

/path/to/Emacs.app/Contents/Emacs -nw

Or for GUI launched via Finder, try adding this at the very beginning of your early-init.el or init.el file:

(setenv "LIBRARY_PATH" "/usr/local/opt/gcc/lib/gcc/10:/usr/local/opt/libgccjit/lib/gcc/10:/usr/local/opt/gcc/lib/gcc/10/gcc/x86_64-apple-darwin19/10.2.0")

You can verify what environment variables the Emacs process itself has with:

M-x describe-variable RET process-environment

[seanfarley]

Ah, ok, this is the error I'm seeing too. It seems that Big Sur (NB: I accidentally typed "Bug Sir" at first) changed the location of some libraries.

Here is the test file from Emac's configure:

$ cat conftest.c

#include 
    #include 
    #include 
    int
    main (int argc, char **argv)
    {
      gcc_jit_context *ctxt;
      gcc_jit_result *result;
      ctxt = gcc_jit_context_acquire ();
      if (!ctxt)
        exit (1);
      gcc_jit_type *int_type =
        gcc_jit_context_get_type (ctxt, GCC_JIT_TYPE_INT);
      gcc_jit_function *func =
        gcc_jit_context_new_function (ctxt, NULL,
                                      GCC_JIT_FUNCTION_EXPORTED,
                                      int_type, "foo", 0, NULL, 0);
      gcc_jit_block *block = gcc_jit_function_new_block (func, "foo");
      gcc_jit_block_end_with_return (
        block,
        NULL,
        gcc_jit_context_new_rvalue_from_int (ctxt, int_type, 1));
      result = gcc_jit_context_compile (ctxt);
      if (!result)
        exit (1);
      typedef int (*fn_type) (void);
      fn_type foo =
        (fn_type)gcc_jit_result_get_code (result, "foo");
      if (!foo)
        exit (1);
      if (foo () != 1)
        exit (1);
      gcc_jit_context_release (ctxt);
      gcc_jit_result_release (result);
      return 0;
    }

Compile it:

$ gcc -o conftest -I/usr/local/include -L/usr/local/lib/gcc/10  conftest.c -lgccjit

Run it:

$ ./conftest
ld: library not found for -lSystem
collect2: error: ld returned 1 exit status
libgccjit.so: error: error invoking gcc driver

Append the new Big Sur location:

$ env LIBRARY_PATH="/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib" ./conftest
$ echo $?
0

[d12frosted]

🤔 Yesterday I tested with GUI, I just run emacs -Q and it loaded just fine. Didn't test how it builds my own lisp files yet. Now I tried loading emacs -Q -nw and I see the same compilation warnings as you folks do.

It seems like GUI doesn't trigger any builds 🤔 And there are no logs in *Messags* buffer.

[d12frosted]

@jimeh tried your hacky patch. Doesn't work for me because it assumes that gcc is located in Emacs.app/Contents/lib. Though it is an interesting approach. I will need to check how you get gcc there 😸 Maybe we can do it here as well.

Without hacky patch, I checked compilation logs in the -nw instance to see these errors:

ld: library not found for -lSystem
collect2: error: ld returned 1 exit status
libgccjit.so: error: error invoking gcc driver
Debugger entered--Lisp error: (native-ice "/usr/local/Cellar/emacs-plus@28/28.0.50/share/emac..." "failed to compile" "/Users\
/d12frosted/.emacs.d/eln-cache/28.0.50-x86_6..." "error invoking gcc driver")
  signal(native-ice ("/usr/local/Cellar/emacs-plus@28/28.0.50/share/emac..." "failed to compile" "/Users/d12frosted/.emacs.d/\
eln-cache/28.0.50-x86_6..." "error invoking gcc driver"))
  comp--native-compile("/usr/local/Cellar/emacs-plus@28/28.0.50/share/emac..." t)
  eval-buffer(#<buffer  *load*> nil "/var/folders/8f/4rf99dhs1p9g6k4bcy8n9w6c0000gn/T/e..." nil t)  ; Reading at buffer posit\
ion 2275
  load-with-code-conversion("/var/folders/8f/4rf99dhs1p9g6k4bcy8n9w6c0000gn/T/e..." "/var/folders/8f/4rf99dhs1p9g6k4bcy8n9w6c\
0000gn/T/e..." nil t)
  command-line-1(("-l" "/var/folders/8f/4rf99dhs1p9g6k4bcy8n9w6c0000gn/T/e..."))
  command-line()
  normal-top-level()

@seanfarley Will try your trick with LIBRARY_PATH="/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib", but now that path is missing 🤔 Apparently I need to upgrade to the latest Xcode and CLT.

[jimeh]

@d12frosted yeah my hacky patch is kinda special... lol

Once I've woken up and had some coffee though I can put together a variant that'll use paths from where homebrew installs gcc and libgccjit.

In the meantime, did you try setting LIBRARY_PATH as early as possible in your config?

(setenv "LIBRARY_PATH" "/usr/local/opt/gcc/lib/gcc/10:/usr/local/opt/libgccjit/lib/gcc/10:/usr/local/opt/gcc/lib/gcc/10/gcc/x86_64-apple-darwin19/10.2.0")

If that doesn't work, I doubt my hacky patch method will work.

[wd]

I have a similar error, fixed by run emacs -Q first, and run load-file ~/.emacs.d/init.el, emacs will load and compile some of the codes, then generate a eln-cache directory.

[d12frosted]

@jimeh

Setting LIBRARY_PATH at the top of init.el doesn't work.

(setenv "LIBRARY_PATH" "/usr/local/opt/gcc/lib/gcc/10:/usr/local/opt/libgccjit/lib/gcc/10:/usr/local/opt/gcc/lib/gcc/10/gcc/x86_64-apple-darwin19/10.2.0")

Still the same error about missing -lSystem.

[jimeh]

@d12frosted that's a shame, and I just finished the absolute path hacky patch 4 seconds before I got a notification about your comment... lol

I'm installing the formula locally to try my luck :)

native-comp-env-setup-absolute.patch

```diff diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el index 25e2de9..bcedd31 100644 --- a/lisp/emacs-lisp/comp.el +++ b/lisp/emacs-lisp/comp.el @@ -2801,6 +2801,56 @@ queued with LOAD %" (comp-run-async-workers) (message "Compilation started.")))) +;;;###autoload +(defun native-compile-setup-environment-variables (&rest _args) + "Ensure LIBRARY_PATH is set correctly when libgccjit is bundled." + (when (eq system-type 'darwin) + (let* ((library-path-env (getenv "LIBRARY_PATH")) + (gccjit-base-dir "/usr/local/opt/libgccjit/lib/gcc") + (gccjit-version (car (seq-filter + (lambda (dir) (string-match-p "^[0-9]+$" dir)) + (directory-files gccjit-base-dir)))) + (gccjit-dir (concat gccjit-base-dir "/" gccjit-version)) + (gcc-base-dir "/usr/local/opt/gcc/lib/gcc") + (gcc-version (car (seq-filter + (lambda (dir) (string-match-p "^[0-9]+$" dir)) + (directory-files gcc-base-dir)))) + (gcc-dir (concat gcc-base-dir "/" gcc-version)) + (darwin-base-dir (car (file-expand-wildcards + (concat gcc-dir "/gcc/*apple-darwin*")))) + (darwin-version (car (seq-filter + (lambda (dir) + (string-match-p + "^[0-9]+\\(\.[0-9]+\\(\.[0-9]+\\)?\\)?$" dir)) + (directory-files darwin-base-dir)))) + (darwin-dir (concat darwin-base-dir "/" darwin-version)) + (lib-paths (append + (list gccjit-dir gcc-dir darwin-dir) + (if library-path-env (list library-path-env) (list))))) + + (when (and gccjit-dir gcc-dir darwin-dir) + (setenv "LIBRARY_PATH" (mapconcat 'identity lib-paths ":"))))) + + ;; Remove advice, as it only needs to run once. + (advice-remove 'native-compile + 'native-compile-setup-environment-variables) + (advice-remove 'comp--native-compile + 'native-compile-setup-environment-variables) + (advice-remove 'native-compile-async + 'native-compile-setup-environment-variables) + (advice-remove 'native--compile-async + 'native-compile-setup-environment-variables)) + +;; Ensure environment setup runs before any native compilation. +(advice-add 'native-compile :before + 'native-compile-setup-environment-variables) +(advice-add 'comp--native-compile :before + 'native-compile-setup-environment-variables) +(advice-add 'native-compile-async :before + 'native-compile-setup-environment-variables) +(advice-add 'native--compile-async :before + 'native-compile-setup-environment-variables) + (provide 'comp) ;;; comp.el ends here ```