Doom totally broken after upgrade (many errors of type " Failed to parse package XXX: use-package: Keyword handler not defined: use-package-handler/:bind")

liar666 commented 1 year ago

I confirm that...

[X] I have searched the issue tracker, documentation, FAQ, Discourse, and Google, in case this issue has already been reported/resolved.
[X] I have read "How to Debug Issues", and will use it to provide as much information about this issue as possible.
[X] The issue can be reproduced on the latest available commit of Doom.
[X] The issue can be reproduced on a stable release of Emacs, such as 27 or 28. (Doom does not support development builds like 29+ or any version ending in .50 or .9x)

Expected behavior

I had a working config and ran an upgrade.

I expected Doom to work exactly the same after my upgrade.

Current behavior

Every time I open DoomEmacs, I now get an error buffer containing:

Error (use-package): Failed to parse package consult-dir: use-package: Keyword handler not defined: use-package-handler/:bind Disable showing Disable logging
Error (use-package): Failed to parse package shift-number: use-package: Keyword handler not defined: use-package-handler/:bind Disable showing Disable logging
Error (use-package): Failed to parse package edwina: use-package: Keyword handler not defined: use-package-handler/:bind Disable showing Disable logging
Error (use-package): Failed to parse package company: use-package: Keyword handler not defined: use-package-handler/:bind Disable showing Disable logging
Error (use-package): Failed to parse package ess-mode: use-package: Keyword handler not defined: use-package-handler/:bind Disable showing Disable logging
Error (use-package): Failed to parse package flutter: use-package: Keyword handler not defined: use-package-handler/:bind Disable showing Disable logging
Error (use-package): Failed to parse package rjsx-mode: use-package: Keyword handler not defined: use-package-handler/:bind Disable showing Disable logging
Error (use-package): Failed to parse package python: use-package: Keyword handler not defined: use-package-handler/:bind Disable showing Disable logging
Error (use-package): Failed to parse package rustic: use-package: Keyword handler not defined: use-package-handler/:bind Disable showing Disable logging
Error (use-package): Failed to parse package julia-mode: use-package: Keyword handler not defined: use-package-handler/:bind Disable showing Disable logging
Error (use-package): Failed to parse package whisper: use-package: Keyword handler not defined: use-package-handler/:bind Disable showing Disable logging
Error (use-package): Failed to parse package read-aloud: use-package: Keyword handler not defined: use-package-handler/:bind Disable showing Disable logging
Error (use-package): Failed to parse package aichat: use-package: Keyword handler not defined: use-package-handler/:bind Disable showing Disable logging

and packages are only partially loaded (particularly key bindings are not loaded)

This seems related to: https://github.com/doomemacs/doomemacs/issues/3865 which was never solved...

Also it is related to https://github.com/jwiegley/use-package/issues/555 which gives a partial workaround: insert a (require 'use-package) at the beginning of the config file. Adding this line at the beginning of my config.org partially works: now when I open DoomEamcs, I get an error buffer with only:

Error (use-package): Failed to parse package consult-dir: use-package: Keyword handler not defined: use-package-handler/:bind Disable showing Disable logging

NOTE: consult-dir is the only package that I did not install myself & comes directly with Doom (or as a dependancy). All the other packages are installed by myself through packages.el and configured with config.org

So apparently the use of use-package is broken in the latest DoomEmacs. And there lack a (require 'use-package) somewhere, notably BEFORE loading of consult-dir.

Steps to reproduce

Get a working config.org with a few packages that use :bind directive.
Run an upgrade.

System Information

https://pastebin.com/EmuU5sr4

liar666 commented 1 year ago

Re-installing Doom From scratch seems to have work...

Not the easiest/fatest way, but I needed a functional Emacs ASAP :)

liar666 commented 1 year ago

I'm still trying to rebuild my config from scratch, trying to find where are the problems that make it nonfunctional most of the time. The problem describe here happened again on a fresh install after I did this:

Tried & failed to make keycast work from package.el (see https://github.com/doomemacs/doomemacs/issues/7353) (NOTE: works if I use (straight-use-package 'keycast) directly in config.org though)
doom clean
doom purge
doom build

After these commands, I get the error EVERY TIME I start DoomEmacs, whatever the status of keycast

Seems to be solved by reinstalling everything. i.e. doing:

\rm -fr ~/.emacs.d/.local
doom sync
M-x lsp-install server
... NOT the most convenient way to solve a problem... Very slow.... Very bandwidth demanding....

Mekk commented 1 year ago

I also got some such errors after doom upgrade (after doing rather large upgrade, previous time I did it in Feb or Mar, in both cases emacs 28.1):

Error (use-package): Failed to parse package crux: use-package: Keyword handler not defined: use-package-handler/:bind Disable showing Disable logging
Error (use-package): Failed to parse package indent-tools: use-package: Keyword handler not defined: use-package-handler/:bind Disable showing Disable logging

(those are packages for which i used :bind in my config).

Solution

Looks like I found less invasive solution (somewhat slow but not bandwith consuming):

$ cd doom-emacs/.local/straight             
$ rm -rf build-28.1 build-28.1-cache.el
$ doom sync

doom sync recompiled all packages (but without the need to redownload them), and once it was done (took about 6 minutes on my machine) emacs started properly and stopped reporting those errors. Mayhaps it would suffice to remove only some subdirectory/ies of build-28.1.

PS While looking for the solution earlier I also removed .local/straight/repos/use-package (so doom sync redownloaded this package). It did not help by itself, but maybe mattered.

Mekk commented 1 year ago

So, in general I suppose that this issue is somewhat related to insufficient recompilation (my bet is that either use-package itself, or something else, for some reason wasn't recompiled in spite of being updated, maybe due to some trickery between git timestamps and filesystem timestamps). I never digged into internals of how doom decides which packages need to be recompiled and why but maybe somebody with knowledge of this process could consider deeper look.

IIRC I already had similar case once or twice some time ago. It may matter that:

I doom upgrade not-too-often (usually 2-4 times a year), so the upgrade is usually big
my upgrades are usually suspended many times (by questions what to do with changed git branches or urls, which I don't always spot immediately) and usually fail a few times (due to some git trouble which require removing some specific repo so it is refetched on retry, for example when some repo dir is considered dirty or some download fails due to obsolete url in remotes or some repo is too divergent for some reason, in general I resolve those by removing related subdir of .local/straight/repos and rerunning doom upgrade)

So one of the scenarios I imagine may be happening is (but it is just wild guess):

doom upgrade fails in the middle due to some git trouble on some repo, some repos are refreshed, some are not
retried doom upgrade recompiles something which is in invalid state (repo not yet refreshed, or maybe some dependency not yet refreshed) giving resulting elc „fresh” timestamps but creating them from not-current code
once everything is properly updated package is not recompiled as elc's timestamps suggest it is already up to date

Still: as I said earlier, removing compiled files and rebuilding from scratch helps and costs only a few minutes of CPU time.

Jdogzz commented 1 year ago

@liar666 @Mekk Are you still able to reproduce these broken doom upgrade runs? If it does happen again, would you please post the output of doom doctor after the breakage occurs and before any attempts to troubleshoot?

Mekk commented 1 year ago

Unfortunately no – I cleaned up and upgraded recently, and I don't have backup of pre-upgrade state.

As I said, it seems to me that problematic scenario is that of huge doom upgrade intermixed with some problems during package refresh. Mayhaps one could try to reproduce it by installing old version of doom (as I said, in my last case „before” state was from February or March), then attempting upgrade to current (I upgraded to mid-August), and provoking some error during some package git update (in general some problem is likely to happen by itself because of branches and url changes, but deliberately making some package repo dirty or killing some git command may make it easier). Not sure whether old version is still installable, though (some package urls disappeared IIRC), and what is the exact trigger. So it may be uneasy.

It seems likely that the problem is somehow related to incorrect .elc files, so maybe it would be easier to approach this by tracking of when critical items (use-package code? bind-key code? some dependency?) are rebuilt during upgrade, especially interrupted and retried upgrade. I do not understand the very error we saw here, but I can imagine for example, that we have two related packages which both evolved and that sth like „use-package compiled with old bind-key, then used with new bind-key without recompilation” may be happening.

If I face similar problem in the future, I will try to doom doctor it. But most likely it won't happen soon.

liar666 commented 10 months ago

Hi,

I had the same problem at least twice. Unfortunately I did the rm -fr straight/build* trick before thinking of running doom doctor... Sorry. I'll try to remember about doing it next time...

For what is worth, I have a few packages that DoomEmacs do not manage to get automatically from its downloading sources when I list them in my init.el file, even if they are supposed to be in the repos (why?!? ; example: shift-number, keycast, flymake-languagetool, aichat). This lead to errors when I configure them in my config.org because the packages are not loaded and the configured variables/functions are not found. As a consequence, I forced install them with commands like (straight-use-package 'keycast) in config.org. I'm sure it's not the correct way of doing things, but it's the only one that worked. Could it be related to the error we see here?

GMTSE commented 10 months ago

OK... Although I did the big cleanup yesterday:

$ rm -rf build-28.1 build-28.1-cache.el
$ doom sync

The problem happened again today.

More or less what I did before it happens:

doom upgrade
doom clean
doom sync # -u ?
doom purge
doom build
doom clean
doom sync

Also, something important to note: packages keycast, shift-number, yatemplate and gscholar-bibtex do not install automatically when I list them in config.org, thus I added calls to (straight-use-package '<package>) right before the (use-pacakge <package> :config) in my config.org.

Other thing to note: right after the doom * commands, I ran a few times \emacs without internet connection. Due to my use of straight, first starts of Emacs right after the doom * commands need to install a few packages (see note above) and these failed with an error saying git can not be reached. I run \emacs again after restoring the internet connection, but that might also have introduced errors?

Here is the output of doom doctor right after the doom * commands and opening of \emacs that first showed the use-package error again: doom_doctor_output.txt And here is the error log referred into the doom doctor output: cli.doom.231127232010.508630.error.txt

GMTSE commented 10 months ago

Happened (AGAIN!!) today... Seems that running doom build is the root cause of the problem, since I did a doom build then a doom sync and some M-x doom/reload. I also did several pkill emacs (probably between the 2 previous commands), because doom sync/reload does not always play right with emacsclient.

hlissner commented 1 month ago

because doom sync/reload does not always play right with emacsclient.

You need to restart the daemon too, after doom sync.

In any case, I'll close this issue because the original issue has to do with packages that failed to clone correctly, which should be mitigated by 1fa8d3a4b99275a1bad2aba8e9195c282b49b12d, at least until v3 is done. Let me know if the issue persists. Thanks for bringing it to my attention!

doomemacs / doomemacs