Remove undo-tree dependency

[alienbogart]

Issue

Undo tree is an undo package that cannot keep undos.

Environment

Emacs version: Emacs 26.1.50 Operating System: Manjaro (current) Evil version: Evil version 1.2.13 Evil installation type: MELP Graphical/Terminal: Graphical Tested in a make emacs session (see CONTRIBUTING.md): Yes Undo configurations:

(use-package undo-tree
  :ensure t
  :init
  (setq undo-limit 78643200)
  (setq undo-outer-limit 104857600)
  (setq undo-strong-limit 157286400)
  (setq undo-tree-mode-lighter " UN")
  (setq undo-tree-auto-save-history t)
  (setq undo-tree-enable-undo-in-region nil)
  (setq undo-tree-history-directory-alist '(("." . "~/emacs.d/undo")))
 (add-hook 'undo-tree-visualizer-mode-hook (lambda ()
                                              (undo-tree-visualizer-selection-mode)
                                              (setq display-line-numbers nil)))
  :config
  (global-undo-tree-mode 1))

Reproduction steps

• Start Emacs
• Make 30 changes on a files an save each step
• Restart Emacs
• Try undoing and redoing those changes

Expected behavior

You should be able to undo and/or redo every single change

Actual behavior

When it works, I'm able to undo 5 changes at the most. Half the time I can't undo anything at all. Sometimes I get the message unrecognized entry in undo list undo-tree-canary while I'm performing the undo.

Further notes

It is not just me:

https://old.reddit.com/r/emacs/comments/85t95p/undo_tree_unrecognized_entry_in_undo_list/ https://debbugs.gnu.org/cgi/bugreport.cgi?bug=16523 https://debbugs.gnu.org/cgi/bugreport.cgi?bug=16377 https://lists.ourproject.org/pipermail/implementations-list/2014-November/002091.html https://github.com/syl20bnr/spacemacs/issues/298 https://github.com/syl20bnr/spacemacs/issues/9903

I also must add that I had this same problem before with entirely different configurations, other versions of Emacs and operating systems.

[duianto]

I haven't wrapped my head around what the expected behaviour of the two sequential C-.'s is.

I'm not sure what's expected either.

The reproduction steps were just reduced to those, after reading your comment that evil-repeat-pop could be a possible source, and encountering the error message while testing it.

---

The last line in the fix is outside the let.

(when undo-disabled (setq buffer-undo-list t))

Because pasting shows:

unwind-protect: Symbol’s value as variable is void: undo-disabled

Moving it inside the let fixes that void issue, but the reproduction steps error remains:

primitive-undo: Unrecognized entry in undo list undo-tree-canary

However now the buffer-undo-list says:

Its value is (nil undo-tree-canary)

[tsc25]

The last line is intentionally outside the let, so it gets called from the unwind-protect. If body throws an error, we still want it to clean up and reset buffer-undo-tree correctly. It's the let that should be outside the unwind-protect, I think:

(defmacro evil-with-undo (&rest body)
  "Execute BODY with enabled undo.
If undo is disabled in the current buffer, the undo information
is discarded."
  `(let ((undo-disabled (eq buffer-undo-list t)))
     (unwind-protect
         (when undo-disabled (setq buffer-undo-list nil))
       (unwind-protect
           (progn ,@body)
         (unless (null (car-safe buffer-undo-list))
           (push nil buffer-undo-list))))
     (when undo-disabled (setq buffer-undo-list t))))

This fixes the error in your example for me. I wonder if the new code is being called in your testing? Because evil-with-undo is a macro that gets invoked by a whole chain of macros, you need to eval or compile the new macro, then re-eval (in this order) evil-with-single-undo and evil-repeat to ensure the new version of evil-with-undo gets used. (Forking evil, updating evil-with-undo and recompiling everything might be simpler!)

[duianto]

The last line is intentionally outside the let, so it gets called from the unwind-protect.

👍

The latest version doesn't show the void message.

---

I replaced the evil-with-undo macro in: https://github.com/emacs-evil/evil/blob/6316dae58e95fe019fcb41d2d1ca5847227a16e6/evil-common.el#L3523

Removed all .elc files from: C:\Users\username\AppData\Roaming\.custom.emacs.d\elpa\evil-20210109.807

And restarted Emacs.

The message remains:

primitive-undo: Unrecognized entry in undo list undo-tree-canary

It didn't help to recompile the evil directory: C-u M-x byte-recompile-directory

Note:

Adding C-u 0 before M-x byte-recompile-directory, recompiles all files without asking for confirmation on every file.

source: the comment: https://stackoverflow.com/a/1217249 in: https://stackoverflow.com/questions/1217180/how-do-i-byte-compile-everything-in-my-emacs-d-directory

[duianto]

It doesn't seem to be related to my custom setup. The Unrecognized message also appears with the new macro, in Manjaro with emacs in the default location: /home/username/.emacs.d/

And without the call to the local mode hook. (add-hook 'evil-local-mode-hook 'turn-on-undo-tree-mode)

It isn't needed in the scratch buffer.

But undo-tree doesn't become enabled in a new buffer without it: C-x b test RET

It's suggested in this comment: https://github.com/emacs-evil/evil/issues/1382#issuecomment-761087738 in the issue: u and C-r doesn't work in new buffer with undo-tree as evil-undo-system #1382

.emacs

(require 'package)
(add-to-list 'package-archives '("melpa" . "https://melpa.org/packages/") t)
;; Comment/uncomment this line to enable MELPA Stable if desired.  See `package-archive-priorities`
;; and `package-pinned-packages`. Most users will not need or want to do this.
;;(add-to-list 'package-archives '("melpa-stable" . "https://stable.melpa.org/packages/") t)
(package-initialize)

(require 'undo-tree)
(global-undo-tree-mode)

(require 'evil)
(evil-mode 1)
(evil-set-undo-system 'undo-tree)

GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.22, cairo version 1.17.3) of 2020-08-28

[tsc25]

Thanks for your efforts @duianto!

Would be good to double-check the replacement macro is being called. I think just recompiling everything won't work, because when evil-common gets recompiled, its version of evil-with-undo will clobber the replacement version, and evil-repeat will use that one when it gets compiled.

As a more fail-safe test, try saving the following in a file, say, "evil-with-undo-test.el", then byte-compile and load it:

(defmacro evil-with-undo (&rest body)
  "Execute BODY with enabled undo.
    If undo is disabled in the current buffer, the undo information
    is discarded."
  `(let ((undo-disabled (eq buffer-undo-list t)))
     (message "In replacement `evil-with-undo'")
     (unwind-protect
         (when undo-disabled (setq buffer-undo-list nil))
       (unwind-protect
           (progn ,@body)
         (unless (null (car-safe buffer-undo-list))
           (push nil buffer-undo-list))))
     (when undo-disabled (setq buffer-undo-list t))))

(defmacro evil-with-single-undo (&rest body)
  "Execute BODY as a single undo step."
  (declare (indent defun)
           (debug t))
  `(let (evil-undo-list-pointer)
     (evil-with-undo
       (unwind-protect
           (progn
             (evil-start-undo-step)
             (let ((evil-in-single-undo t))
               ,@body))
         (evil-end-undo-step)))))

(defun evil-repeat (count &optional save-point)
  "Repeat the last editing command with count replaced by COUNT.
If SAVE-POINT is non-nil, do not move point."
  (interactive (list current-prefix-arg
                     (not evil-repeat-move-cursor)))
  (cond
   ((null evil-repeat-ring)
    (error "Already executing repeat"))
   (save-point
    (save-excursion
      (evil-repeat count)))
   (t
    (unwind-protect
        (let ((confirm-kill-emacs t)
              (kill-buffer-hook
               (cons #'(lambda ()
                         (user-error "Cannot delete buffer in repeat command"))
                     kill-buffer-hook))
              (undo-pointer buffer-undo-list))
          (evil-with-single-undo
            (setq evil-last-repeat (list (point) count undo-pointer))
            (evil-execute-repeat-info-with-count
             count (ring-ref evil-repeat-ring 0))))
      (evil-normal-state)))))

When you run your recipe for reproducing the error, you should see the "In replacement `evil-with-undo'" message in the *Messages* buffer. If it's not there, it means the new macro isn't being used.

If it is there, confirming the new macro is being used, and you still see the error, then I'll have to figure out how to reproduce this new case. Just tested again following the above procedure, and the error is definitely gone on my system.

[tsc25]

Ah, occurs to me that I had also made some local changes to undo-tree.el whilst testing yesterday. I just pushed these to Gitlab, in case they make a difference here. (undo-tree.el version number 0.8.1).

[duianto]

I copied the contents of the current file: undo-tree.el (0.8.1) https://gitlab.com/tsc25/undo-tree/-/blob/cbe0c708d8a71a521199bd8e3e605c760ecdb021/undo-tree.el

Overwrote the local file: C:\Users\username\AppData\Roaming\.custom.emacs.d\elpa\undo-tree-0.7.5\undo-tree.el

Removed: undo-tree.elc

Created a new file in: C:\Users\username\AppData\Roaming\.custom.emacs.d\evil-with-undo-test.el with the versions of the three functions above https://github.com/emacs-evil/evil/issues/1074#issuecomment-780067369:

evil-with-undo
evil-with-single-undo
evil-repeat

Started Emacs.

It complained that it couldn't find queue. Checked the difference between the previous and current undo-tree.el

Saw that it now says:

;; Package-Requires: ((queue "0.2"))

Installed the package: M-x package-install queue RET

Restarted Emacs.

M-x load-file C:\Users\username\AppData\Roaming\.custom.emacs.d\evil-with-undo-test.el RET

C-h f evil-with-undo RET shows:

evil-with-undo is a Lisp macro in ‘../../evil-with-undo-test.el’.

(evil-with-undo &rest BODY)

Execute BODY with enabled undo.
    If undo is disabled in the current buffer, the undo information
    is discarded.

[back]

When following the reproduction steps: Pressing . (period) shows:

In replacement ‘evil-with-undo’

The first C-. shows:

Undo In replacement ‘evil-with-undo’ Undo branch point!

The second C-. shows:

primitive-undo: Unrecognized entry in undo list undo-tree-canary

The buffer-undo-list has the value:

Its value is (nil undo-tree-canary)

The same thing is observed with or without calling byte-compile-file for the files: undo-tree.el evil-with-undo-test.el

[tsc25]

Thanks @duianto - I can reproduce this now.

The macro change fixed one bug: buffer-undo-list isn't getting corrupted anymore. Now it hits the second bug I described in my long message above: evil-undo-pop (which gets called during evil-repeat-pop) tries to undo the last buffer modification by calling the vanilla Emacs undo function. Which is simply wrong in undo-tree-mode, since that undo history has been transferred to buffer-undo-tree.

To fix this one, I'll have to figure out what evil-undo-pop is doing, and undo the appropriate changes using undo-tree-mode commands instead of vanilla Emacs undo...

[Hi-Angel]

The macro change fixed one bug: buffer-undo-list isn't getting corrupted anymore. Now it hits the second bug I described in my long message above: evil-undo-pop (which gets called during evil-repeat-pop) tries to undo the last buffer modification by calling the vanilla Emacs undo function. Which is simply wrong in undo-tree-mode, since that undo history has been transferred to buffer-undo-tree.

Hmm, on a side note, I think it might be worth making undo-tree warn about that or something… The thing is, I don't think evil-mode is the only one who might try to "vanilla-undo". In fact, right now as I'm writing the text, I made use of C+/ which calls vanilla undo, to remove last word with a typo without entering "insert-mode".

I realize I gotta rebind the combination to the appropriate function of undo-tree, but my point is that there might be other users of the vanilla undo who wouldn't know about it, and they will likely get hit by such conflict between the two undo systems.

[Hi-Angel]

I realize I gotta rebind the combination to the appropriate function of undo-tree, but my point is that there might be other users of the vanilla undo who wouldn't know about it, and they will likely get hit by such conflict between the two undo systems.

Perhaps undo-tree mode could override the undo function?

[tsc25]

I realize I gotta rebind the combination to the appropriate function of undo-tree, but my point is that there might be other users of the vanilla undo who wouldn't know about it, and they will likely get hit by such conflict between the two undo systems.

You don't have to rebind anything to use undo-tree: it installs a command remapping from undo (and various others) to the appropriate undo-tree functions. Any interactive calls to undo (and related commands) in undo-tree-mode will do the right thing. (See command remapping in the Elisp manual. Emacs has had this mechanism since before the undo-tree package existed.)

Perhaps undo-tree mode could override the undo function?

It already does override it, and always has done, using the appropriate Emacs mechanism (command remapping). The undo-tree package can't redefine the undo function itself, even if that was a sensible thing to do in a package (it isn't). The undo function is still needed in buffers where undo-tree-mode is not enabled, and potentially by low-level Emacs code.

The only(*) way to call the vanilla undo function in undo-tree-mode is from Elisp code. And Elisp code is expected to know what it's doing: there are a gazillion ways to shoot oneself in the foot from Elisp!

(*) Unless you go to strenuous efforts to bind undo in a keymap that overrides minor-mode maps. And that also requires Elisp.

To be clear, Evil's evil-undo and evil-redo commands already call the correct functions, depending on the setting of evil-undo-system. The bug is due to Evil calling undo directly from a low-level, auxiliary function (evil-undo-pop). That will only work correctly with the vanilla Emacs undo system; doing it in undo-tree-mode is a bug, and the Evil code ought to call the appropriate function here depending on the setting of evil-undo-system, as it does with the evil-undo command.

But the buggy implementation will work OK most of the time. It takes a subtle combination of circumstances to trigger the bug: you need an undo-tree command to get called just before evil-undo-pop is invoked. Normally, undo-tree commands only get called when the user calls them interactively (via evil-undo or evil-redo), and evil-undo-pop doesn't get called in that case. @duianto's recipe gets evil-undo into the Evil repeat ring, which sets it up to get called implicitly in the middle of processing the evil-repeat-pop action.

It's "just" a bug that needs fixing in evil-mode.

[Hi-Angel]

It already does override it, and always has done, using the appropriate Emacs mechanism (command remapping). The undo-tree package can't redefine the undo function itself, even if that was a sensible thing to do in a package (it isn't). The undo function is still needed in buffers where undo-tree-mode is not enabled, and potentially by low-level Emacs code.

The only(*) way to call the vanilla undo function in undo-tree-mode is from Elisp code, as Evil is doing. And Elisp code is expected to know what it's doing: there are a gazillion ways to shoot oneself in the foot from Elisp!

Ah, I see, thank you for elaboration

[tsc25]

To fix this one, I'll have to figure out what evil-undo-pop is doing, and undo the appropriate changes using undo-tree-mode commands instead of vanilla Emacs undo.

OK, I think this might be as simple as making undo-tree-pop just call undo-tree-undo when undo-tree is used (together with the evil-with-undo macro fix from above).

Same proviso as before: I'm not certain what the expected behaviour should be. But loading the following code at least makes the "Unrecognized entry" error go away for me in @duianto's example:

(defmacro evil-with-undo (&rest body)
  "Execute BODY with enabled undo.
        If undo is disabled in the current buffer, the undo information
        is discarded."
  `(let ((undo-disabled (eq buffer-undo-list t)))
     (unwind-protect
         (when undo-disabled (setq buffer-undo-list nil))
       (unwind-protect
           (progn ,@body)
         (unless (null (car-safe buffer-undo-list))
           (push nil buffer-undo-list))))
     (when undo-disabled (setq buffer-undo-list t))))

(defmacro evil-with-single-undo (&rest body)
  "Execute BODY as a single undo step."
  (declare (indent defun)
           (debug t))
  `(let (evil-undo-list-pointer)
     (evil-with-undo
       (unwind-protect
           (progn
             (evil-start-undo-step)
             (let ((evil-in-single-undo t))
               ,@body))
         (evil-end-undo-step)))))

(defun evil-repeat (count &optional save-point)
  "Repeat the last editing command with count replaced by COUNT.
    If SAVE-POINT is non-nil, do not move point."
  (interactive (list current-prefix-arg
                     (not evil-repeat-move-cursor)))
  (cond
   ((null evil-repeat-ring)
    (error "Already executing repeat"))
   (save-point
    (save-excursion
      (evil-repeat count)))
   (t
    (unwind-protect
        (let ((confirm-kill-emacs t)
              (kill-buffer-hook
               (cons #'(lambda ()
                         (user-error "Cannot delete buffer in repeat command"))
                     kill-buffer-hook))
              (undo-pointer buffer-undo-list))
          (evil-with-single-undo
            (setq evil-last-repeat (list (point) count undo-pointer))
            (evil-execute-repeat-info-with-count
             count (ring-ref evil-repeat-ring 0))))
      (evil-normal-state)))))

(defun evil-repeat-pop (count &optional save-point)
  "Replace the just repeated command with a previously executed command.
Only allowed after `evil-repeat', `evil-repeat-pop' or
`evil-repeat-pop-next'. Uses the same repeat count that
was used for the first repeat.

The COUNT argument inserts the COUNT-th previous kill.
If COUNT is negative, this is a more recent kill."
  (interactive (list (prefix-numeric-value current-prefix-arg)
                     (not evil-repeat-move-cursor)))
  (cond
   ((not (and (eq last-command #'evil-repeat)
              evil-last-repeat))
    (user-error "Previous command was not evil-repeat: %s" last-command))
   (save-point
    (save-excursion
      (evil-repeat-pop count)))
   (t
    (unless (eq buffer-undo-list (nth 2 evil-last-repeat))
      (evil-undo-pop))
    (goto-char (car evil-last-repeat))
    ;; rotate the repeat-ring
    (while (> count 0)
      (when evil-repeat-ring
        (ring-insert-at-beginning evil-repeat-ring
                                  (ring-remove evil-repeat-ring 0)))
      (setq count (1- count)))
    (while (< count 0)
      (when evil-repeat-ring
        (ring-insert evil-repeat-ring
                     (ring-remove evil-repeat-ring)))
      (setq count (1+ count)))
    (setq this-command #'evil-repeat)
    (evil-repeat (cadr evil-last-repeat)))))

(defun evil-undo-pop ()
  "Undo the last buffer change.
Removes the last undo information from `buffer-undo-list'.
If undo is disabled in the current buffer, use the information
in `evil-temporary-undo' instead."
  (if (and (eq evil-undo-system 'undo-tree)
           (not (eq buffer-undo-list t)))
      (undo-tree-undo)
    (let ((paste-undo (list nil)))
      (let ((undo-list (if (eq buffer-undo-list t)
                           evil-temporary-undo
                         buffer-undo-list)))
        (when (or (not undo-list) (car undo-list))
          (user-error "Can't undo previous change"))
        (while (and undo-list (null (car undo-list)))
          (pop undo-list)) ; remove nil
        (while (and undo-list (car undo-list))
          (push (pop undo-list) paste-undo))
        (let ((buffer-undo-list (nreverse paste-undo)))
          (evil-save-echo-area
            (undo)))
        (if (eq buffer-undo-list t)
            (setq evil-temporary-undo nil)
          (setq buffer-undo-list undo-list))))))

[duianto]

I'm still getting the Unrecognized message after the second C-. In both Manjaro and Windows. With the latest code block.

buffer-undo-list is a variable defined in ‘C source code’.
Its value is (nil undo-tree-canary)
Local in buffer *scratch*; global value is nil

The contents of evil-with-undo-test.el was replaced, with the latest code block above: https://github.com/emacs-evil/evil/issues/1074#issuecomment-780876055

C:\Users\username\AppData\Roaming\.custom.emacs.d\elpa\undo-tree-0.7.5\undo-tree.el is still overwritten with the contents of: https://gitlab.com/tsc25/undo-tree/-/blob/master/undo-tree.el ;; Version: 0.8.1

The file was loaded on Emacs startup: M-x load-file C:\Users\username\AppData\Roaming\.custom.emacs.d\evil-with-undo-test.el RET

C-h f evil-with-undo shows:

evil-with-undo is a Lisp macro in ‘../../evil-with-undo-test.el’.

(evil-with-undo &rest BODY)

Execute BODY with enabled undo.
        If undo is disabled in the current buffer, the undo information
        is discarded.

[back]

[tsc25]

@duianto I think this is because you're not setting the evil-undo-system variable in your test setup. Normally this variable is set via customize, which calls evil-set-undo-system to set the undo/redo functions to undo-tree's. Calling the evil-set-undo-system function directly doesn't set the evil-undo-system variable itself.

Could you try again, adding

(setq evil-undo-system 'undo-tree)

immediately before the evil-set-undo-system line in your .custom-init.el?

[duianto]

Confirmed.

With only: (evil-set-undo-system 'undo-tree) in the init.el file. The variable evil-undo-system had the value nil on startup.

With:

(setq evil-undo-system 'undo-tree)
(evil-set-undo-system 'undo-tree)

or just this single line:

(custom-set-variables '(evil-undo-system 'undo-tree))

source: u and C-r doesn't work in new buffer with undo-tree as evil-undo-system #1382

Now there is no Unrecognized entry message when pressing C-..

Continuing to press C-. removes the initial scratch buffer text

;; This buffer is for text that is not saved, ...

And after the fourth time C-. is pressed (six in total).

Then the minibuffer shows:

No further undo information

Which most likely is expected.