Automatic byte compilation cause startup and loading org-mode slow

[Eason0210]

Hi @purcell ,

I find that theatuo-compile in init-lisp.el will cost a long time to startup Emacs. And it aslo cause loading org-mode very slow at the first time.(I need to 18s)

;; ----------------------------------------------------------------------------
;; Automatic byte compilation
;; ----------------------------------------------------------------------------
(when (maybe-require-package 'auto-compile)
  (add-hook 'after-init-hook 'auto-compile-on-save-mode)
  (add-hook 'after-init-hook 'auto-compile-on-load-mode))

Emacs 28.0.50 / 27.1 Mac OS / Windows 10

[purcell]

That's strange. It absolutely doesn't have that effect for me, and indeed it only affects saving of elisp files. I suspect you've got something else going on, like outdated or missing .elc files.

[Eason0210]

Thanks for your information. On the below message, you can see it take 21s to restore an org file with only one line of content. I will investigate more on it and find if there are missing .elc files.

When the auto-commpile active:

Loading c:/Users/Aqua/.emacs.d/custom.el (source)...done
Loading c:/Users/Aqua/.emacs.d/.session...done
Restoring clock data
Loading c:/Users/Aqua/.emacs.d/org-clock-save.el (source)...done
Desktop: 21985.08ms to restore ~/org.org
Desktop: 1045.89ms to restore ~/.emacs.d/init.el
Desktop: 801.42ms to restore nil
Wrote c:/Users/Aqua/.emacs.d/.emacs.desktop.lock
Desktop: 1 frame, 3 buffers restored.
Desktop restored in 23878.63ms
Loading c:/Users/Aqua/.emacs.d/recentf...done
Cleaning up the recentf list...done (0 removed)
Saving file c:/Users/Aqua/.emacs.d/custom.el...
Wrote c:/Users/Aqua/.emacs.d/custom.el
init completed in 3438.62ms

When removed auto-compile:

Loading c:/Users/Aqua/.emacs.d/custom.el (source)...done
Loading c:/Users/Aqua/.emacs.d/.session...done
Restoring clock data
Loading c:/Users/Aqua/.emacs.d/org-clock-save.el (source)...done
Desktop: 7704.16ms to restore ~/org.org
Desktop: 385.24ms to restore nil
Wrote c:/Users/Aqua/.emacs.d/.emacs.desktop.lock
Desktop: 1 frame, 2 buffers restored.
Desktop restored in 8125.66ms
Loading c:/Users/Aqua/.emacs.d/recentf...done
Cleaning up the recentf list...done (0 removed)
Saving file c:/Users/Aqua/.emacs.d/custom.el...
Wrote c:/Users/Aqua/.emacs.d/custom.el
init completed in 3384.54ms

The result above tested on Windows 10 , emacs 28.0.50

[purcell]

What's goinmg on in org.org? Does the same thing happen if that file is empty? Perhaps there's lots of stuff getting tangled into little .el files that get automatically byte-compiled because of auto-compile? If so, that's either an issue for auto-compile, or you'll need to find a way to suppress auto-compilation of those files by configuring/overriding auto-compile somehow.

[Eason0210]

The org.org only have one line. And I tested again with an empty org file, and the issue still existed.
I tested the auto-compile with emacs -Q , it do not influence org mode.

[purcell]

Weird. Any guesses at what might be going on, @tarsius?

[Eason0210]

I guess the issue is from org-mode.
With Windows 10 OS system, I tested on Emacs 26.3 , 27.1 and 28.0.50 master branch, Theorg-modeneed 1500ms to load an empty org file. When I add more and more packages with this config, them time for loading org-mode will grow to 10000ms an more. And withauto-compile to activate, the time will double.

[tarsius]

Of course it takes time to recompile many of Org's libraries. Since auto-compile can detect outdated libraries at load-time, that can happen then. Also just loading Org takes some time, so it is not surprising that opening or otherwise dealing with the first Org file is slow, it does not matter whether it is empty. No surprise there.

But in the last message you mention that there is a delay for every Org file, not just the first. I am not sure what "load" means here does it mean find-file or org-babel+load? If it is the latter then auto-compile can contribute to the time being spend but that would still be expected. If you do *.org (tangle)=> *.el (compile)=> *.elc, then of course doing the first step means the second has to be taken as well.

[Eason0210]

Only to open org file at the first time slowly. What I mean “load” is opening an org file by find-file or any other ways, or by org-agenda(C-c, a).

[tarsius]

Then I don't see how auto-compile could possibly cause that.

[purcell]

Strangely enough, I can reproduce this: 17 seconds with auto-compile-on-load enabled, 9.4 seconds without. There's at least a hundred files being loaded in order to bootstrap that first trivial org-mode buffer, so I guess the question is whether ~80ms/file is the expected overhead for checking if the .elc is up to date. It feels a little high.

[purcell]

This isn't particularly an issue for me, I should point out. Mainly a curiosity.

[tarsius]

One could (setq auto-compile-delete-stray-dest nil) to avoid some optional work. Beyond that I don't see how to speed things up.

[purcell]

Hmm, does that affect auto-compile-on-load though?

One thing that occurred to me is that it could potentially be useful to shortcut auto-compile-on-load checks entirely for byte-code that is in the emacs standard library, on the basis that it's been pre-compiled in the process of distribution. Not sure if that's something we can reliably identify.

[Eason0210]

I tried to put (setq org-modules-loaded t) into the early-init.el, and the time to open an org file will be reduced from 10s to 6s with auto-compile-on-load enabled.

You can get more info at about org-modules-loaded for here: https://github.com/bzg/org-mode/commit/edfafa909064e3b4e6533c0b71adbb009b037bcb

Tested on Mac 11.1 Big Sur, Emacs 28 latest master branch

[tarsius]

For each library the complete load-path is searched three times. Twice in auto-compile-on-load and once in require itself.

The former could theoretically be merged into one search but that would not be faster than (setq auto-compile-delete-stray-dest nil), which completely disables the second search. It probably would be slower because we would have to implement it in lisp while the current approach boils down to two calls to locate-file-internal, which is implemented in C.

I also don't see how we could "shortcut auto-compile-on-load checks". To find out whether a certain library is a built-in library we would need to search the load-path, and once we have done that, we already have done the thing we would have liked to avoid, which is to search the load-path.

so I guess the question is whether ~80ms/file is the expected overhead for checking if the .elc is up to date. It feels a little high.

I am afraid that's what it boils down to and the answer is yes.

[purcell]

Right, yeah, it figures that searching the load path would be a major cost.

I also don't see how we could "shortcut auto-compile-on-load checks". To find out whether a certain library is a built-in library we would need to search the load-path, and once we have done that, we already have done the thing we would have liked to avoid, which is to search the load-path.

That makes perfect sense given the above.

[Eason0210]

Now, it is clear that what will cause this issue. I set (setq org-modules-loaded t) on my local early-init.el , and the loading time is acceptable for me. So I close this issue now.

[purcell]

I set (setq org-modules-loaded t) on my local early-init.el

What are the trade-offs of this? Should I make it the default? It's not clear to me what effect it actually has.

[Eason0210]

What are the trade-offs of this?

The trade-offs is that the modules defined in the variable org-modules will not be loaded any more. Because I don't use any one of these modules, so I set (setq org-modules-loaded t) . Users can aslo customize org-modules to get the same affect, only keep one or two modules they like.

(defcustom org-modules '(ol-w3m ol-bbdb ol-bibtex ol-docview ol-gnus ol-info ol-irc ol-mhe ol-rmail ol-eww)

Should I make it the default?

I think we don't need to make it default. It is better to let users customize it by themself, may be there are some users is using these modules.

[purcell]

I think we don't need to make it default. It is better to let users customize it by themself, may be there are some users is using these modules.

Agree, thanks.