rational-compile module always compiles modules, even when no change has been made

[jeffbowman]

The rational-compile module compiles modules on startup each time Emacs starts. It should only compile modules if the source is newer than the elc byte-compiled file.

cc: @abarocio80

[abarocio80]

I am trying to manually verfy if the source file needs compilation, and I learned about native-compile-async-skip-p, but it always answers nil to skip compilation.

Is there any issue or bug with this function? Or am I understanding wrong the functionality/interface of this function?

[jeastman]

It looks like native-compile-async-skip-p checks for a few error conditions (nil selector, etc.) and if the item is currently being compiled. It does not appear to check if the file is newer than the binary.

One fairly easy solution would be to check the compiled version against the source with file-newer-than-file-p.

[abarocio80]

Thanks, @jeastman, for the tip.

I already implemeted such function, but it still tries to compile everything on startup, even what I have already compiled.

I belive, the question is "How do we locate the .eln compiled file from a source file?" My atempt failed.

This is what I added

(defvar rational-compile--eln-load-path '()
  "Diectories to search in the modules native-compiled files.")
;; [...]
(dolist (path native-comp-eln-load-path)
  (when (and (file-exists-p path)
             (or (string-prefix-p user-emacs-directory path)
                 (string-prefix-p rational-config-path path)))
    (dolist (dir (directory-files path))
      (when (string-prefix-p emacs-version dir)
        (add-to-list 'rational-compile--eln-load-path
                     (expand-file-name dir path))))))
;; [...]
(defun rational-compile-locate-compiled (source)
  "Locates the compiled version of SOURCE in
`rational-compile--eln-load-path'."
  (let ((compiled nil)
        (base (file-name-base source)))
    (if (featurep 'native-compile)
        (dolist (dir rational-compile--eln-load-path)
          (when (file-exists-p dir)
            (dolist (eln (directory-files dir))
              (when (and (not compiled)
                         (string-suffix-p ".eln" eln)
                         (string-prefix-p base eln))
                (setq-local compiled (expand-file-name eln dir))))))
      (let ((elc (concat source "c")))
        (when (file-exists-p elc)
          (setq-local compiled elc))))
    compiled))
;; [...]
(defun rational-compile-file-p (source)
  "Whether to compile SOURCE or not."
  (let ((target (rational-compile-locate-compiled source)))
    (if target
        (file-newer-than-file-p source target)
      t)))

;; A function to compile a specific file
(defun rational-compile-file (f)
  "Compiles (native or byte-code) the file F.

F could be a single file or a list of files.

If F is a directory or contains a directory, the content of that
directory will be compiled, but not it's subdirectories."
  (dolist (source (flatten-list `(,f)))
    (when (and (file-exists-p source)
               (rational-compile-file-p source))
      (message "Compiling file: %s" source)
      (if (featurep 'native-compile)
          (native-compile-async source nil t)
        (byte-recompile-file source nil 0)))))

If any one finds the error, or the way to correctly locate the eln files, please, be kind to tell.

[jeffbowman]

Is there a PR forthcoming on this? This issue seems to have stalled...

@abarocio80 @jeastman

[abarocio80]

Sorry I haven't seen the message.

I have been working on that, and I thought I had it solved. But a new bug kicked in (with native-compile).

When you start rational-emacs from just the repo and your config (configurated to use rational-compile) it suddenly doesn't seem to be able to use rational-package-install-package.

I only managed to return it to a sane state by:

1. Deleting all .elc files in the path, and
2. Loading first my emacs session and afterwards (with all loaded), compile the modules and configuration files.

I will make the PR, in case anyone could find what's the bug, or if it is particular to my system.

[Cerebus]

I'm still having this issue with 28.2 native-comp. On every startup, packages are recompiled:

Loading /home/cerebus2/.local/share/crafted-emacs/bootstrap/crafted-package.el (source)...done
Loading /home/cerebus2/.local/share/crafted-emacs/bootstrap/crafted-package-bootstrap.el (source)...done
crafted-init: package archives stale, refreshing in the background
Importing package-keyring.gpg...done
Loading quail/latin-ltx...done
Loading quail/sgml-input...done
Loading quail/rfc1345...done
Contacting host: stable.melpa.org:443
Generating autoloads for py-isort.el...done
  INFO     Scraping files for py-isort-autoloads.el...done
Wrote /home/cerebus2/.config/crafted-emacs/elpa/py-isort-2016.1/py-isort-autoloads.el
Checking /home/cerebus2/.config/crafted-emacs/elpa/py-isort-2016.1... [3 times]
Compiling /home/cerebus2/.config/crafted-emacs/elpa/py-isort-2016.1/py-isort.el...done
Wrote /home/cerebus2/.config/crafted-emacs/elpa/py-isort-2016.1/py-isort.elc
Checking /home/cerebus2/.config/crafted-emacs/elpa/py-isort-2016.1...
Done (Total of 1 file compiled, 2 skipped)
Package ‘py-isort’ installed.
Loading /home/cerebus2/.config/crafted-emacs/custom.el (source)...done
Saving file /home/cerebus2/.config/crafted-emacs/custom.el...
Wrote /home/cerebus2/.config/crafted-emacs/custom.el
Loading /home/cerebus2/.config/crafted-emacs/var/recentf...done
Cleaning up the recentf list...done (0 removed)
For information about GNU Emacs and the GNU system, type C-h C-a.
Loading Crafted Startup Screen
Checking for Crafted Emacs updates...
Crafted Emacs is up to date!
Showing recents on splash screen
Compiling file(s): (/home/cerebus2/.config/crafted-emacs/early-config.el /home/cerebus2/.config/crafted-emacs/config.el)
Crafted Emacs loaded in 2.811903 seconds.
Package refresh done

[jeffbowman]

The observation I have is related to paths. It appears you have some things in your ~/.local/share/crafted-emacs folder, but the compile code is looking for things to compile in ~/.config/crafted-emacs. Packages look to be installed in ~/.config/crafted-emacs as well. You might need to look into how your crafted-config-path is being set. If you can post a link to your dot files, I can also take a look.

If you are compiling source from ~/.config/crafted-emacs and putting the .elc files in ~/.local/share/crafted-emacs, then the source will always appear newer and be compiled. Symlink may also confuse things as the code may not follow symlinks correctly. I've had that problem on a Mac before.

Does any of that help?

[Cerebus]

Using CHEMACS2, so following the README:

 (
    ("default" . ((user-emacs-directory . "~/.local/share/doom-emacs")))
    ("crafted" . (
                  (user-emacs-directory . "~/.local/share/crafted-emacs")
                  (env . (("CRAFTED_EMACS_HOME" . "~/.config/crafted-emacs")))
                  )
     )
  )

I like keeping the local modifications outside of the crafted-emacs source, because I'm controlling my dotfiles with git separately. The module's ELN is going to user-emacs-directory, and load-path is picking up my custom-modules in CRAFTED_EMACS_HOME properly.

Is this an issue specific to crafted-compile and CRAFTED_EMACS_HOME?

[Cerebus]

And weirdly, this only happens on Linux (emacs from source @28.2). The same crafted configuration on macOS (d12frosted/emacs-plus@28.2) doesn't recompile anything.

I'm also getting this in the async compile log (Linux only):

Compiling /home/cerebus2/.config/crafted-emacs/elpa/py-isort-2016.1/py-isort.el...
Compiling /home/cerebus2/.config/crafted-emacs/config.el...
/home/cerebus2/.config/crafted-emacs/config.el: Error: Symbol's value as variable is void crafted-config-var-directory

Which is weird, b/c is is set:

crafted-config-var-directory is a variable defined in init.el.

Value
"/home/cerebus2/.config/crafted-emacs/var/"

[jeffbowman]

Took me a while. I configured my chemacs2 similar to yours in the sense that I have separate folders for where I checkout crafted-emacs vs where my CRAFTED_CONFIG_PATH is, here is what I found.

The logging in *Messages* is misleading.

Here is what it looks like when it is actually compiling the files:

Compiling file(s): (/home/vagrant/crafted/153/early-config.el /home/vagrant/crafted/153/config.el)
Compiling /home/vagrant/crafted/153/config.el...done
Wrote /home/vagrant/crafted/153/config.elc
Crafted Emacs loaded in 0.568229 seconds.

Here is what it looks like when it is not actually compiling the files:

Compiling file(s): (/home/vagrant/crafted/153/early-config.el /home/vagrant/crafted/153/config.el)
Crafted Emacs loaded in 0.569590 seconds.

So, it "says" it is "Compile file(s): ..." however, it is not. That is the list of file which would be candidates to recompile if needed. In my case, I don't have an early-config.el.

Looking back at your *Messages* log, it appears it is not compiling those files. Try pulling up ~/.config/crafted-emacs in dired and check the timestamps on the .elc files which are mentioned. My guess is the date is in the past some time.

Does this help?

[Cerebus]

Actually, it is recompiling py-isort.el on each run:

Contacting host: stable.melpa.org:443
Generating autoloads for py-isort.el...done
  INFO     Scraping files for py-isort-autoloads.el...done
Wrote /home/cerebus2/.config/crafted-emacs/elpa/py-isort-2016.1/py-isort-autoloads.el
Checking /home/cerebus2/.config/crafted-emacs/elpa/py-isort-2016.1... [3 times]
Compiling /home/cerebus2/.config/crafted-emacs/elpa/py-isort-2016.1/py-isort.el...done
Wrote /home/cerebus2/.config/crafted-emacs/elpa/py-isort-2016.1/py-isort.elc
Checking /home/cerebus2/.config/crafted-emacs/elpa/py-isort-2016.1...
Done (Total of 1 file compiled, 2 skipped)

Likely b/c it's installing py-isort on each launch despite it being already installed. I can verify that the elc is actually being update on disk every every time; it't not a spurious report.

[jeffbowman]

Yep, installing a package each time will cause it to compile, that isn't the crafted-compile module doing it though, it is package.el. I ignored that section of the log as it isn't/shouldn't be affected by the crafted-compile module.

Wonder why it is installing it each time you start Emacs though... that seems odd... I'll add that to my test and see if I can replicate it.

---

Well, I'm stumped a bit. There was an issue with packages always installing that I believe was fixed back in August. Your log seems to indicate you have an up-to-date version of crafted-emacs, so you should have the fix. I'm curious, if you were to delete the ~/.local/share/crafted-emacs/elpa if it exists and delete the ~/.config/crafted-emacs/elpa if it exists and then start emacs again, which folder gets the packages installed? Then if you restart emacs, do any packages install again?

[Cerebus]

I'd previously removed the elpa caches, but apparently not have gotten both at the same time.

On restart, user-emacs-directory the elpa archive cache and the ELNs; CRAFTED_CONFIG_PATH elpa gets the downloaded modules and the ELCs. Restarting again, no second download or compilation.

So, solved, but still curious as to why it broke in the first place.

[jeffbowman]

Yeah, no idea. I hypothesize package.el gets confused when it finds 2 folders full of packages. This problem only happens with chemacs2 which is why I had to apply the fix I did.

[Cerebus]

Still trying to chase this down from the async compile log. It only shows up when I require crafted-completion and crafted-compile.

Compiling /home/cerebus2/.config/crafted-emacs/config.el...
/home/cerebus2/.config/crafted-emacs/config.el: Error: Symbol's value as variable is void crafted-config-var-directory

I bisected my entire config to get to that combination.

(ETR the reference to py-isort.el; the error is in re: compiling config.el when crafted-completion and crafted-compile -- and nothing else)

[jeffbowman]

Do you not get a stack trace with that error if you start with --debug-init?

---

I haven't been able to duplicate the error you are seeing, here is my test config.el file:

(require 'crafted-compile)
(require 'crafted-mastering-emacs)
(require 'crafted-completion)

(crafted-package-install-package 'flymake-proselint)
(crafted-package-install-package 'corfu)
(crafted-package-install-package 'ack)
(crafted-package-install-package 'py-isort)

;; this compiles the early-config.el and config.el files, if they
;; exist and need to be compiled
(setq crafted-compile-user-configuration t)

(when (fboundp 'use-icomplete-vertical)
 (use-icomplete-vertical))

[Cerebus]

There's no stack trace, that's part of the problem. I can cause this error with only the compile and completion modules.

[jeffbowman]

Shucks.... can you share a link to your config? Email me directly if you'd rather not post it here.

[Cerebus]

The frustrating thing is this isn't happening on my macOS host, only on my Linux host. 😡 OTOH, my macOS host /always downloads all the archives/... which my Linux host does not (and it ignores the cert trust acceptance each time, too).

I'mna burn down and rebuild. If it still happens I'll open a new ticket.

[sthesing]

I'mna burn down and rebuild. If it still happens I'll open a new ticket.

Closing this, then. Feel free to reopen or open a new ticket if the matter comes up again.