:custom settings being _slightly_ overwritten

[sellout]

tl;dr: (get '<some-variable> 'saved-value) changes after the package defining <some-variable> is loaded, depending on whether use-package :custom or custom-set-variables is used to set <some-variable> in an init file.

This is with

• GNU Emacs 28.1 (build 1, aarch64-apple-darwin21.6.0, NS appkit-2113.60 Version 12.6 (Build 21G115))
• use-package 20220910.25 (Commit: 570bde6b4b89eb74eaf47dda64004cd575f9d953)

The high-level view is that I have this in my init file:

(use-package epg :custom (epg-gpg-program "/path/to/gpg2"))

And when I start Emacs, M-x customize-variable epg-gpg-program shows me that value ("/path/to/gpg2"), but it says “CHANGED outside Customize.” and is missing the “Customized with use-package epg” comment. Then, if I try to use epg, I get an error “epg-context--make: GPG error: "no usable configuration", OpenPGP”.

Digging into that a bit, I update my init file to

(use-package epg
  :custom (epg-gpg-program "/path/to/gpg2")
  :init (print (symbol-plist 'epg-gpg-program)))

(print (symbol-plist 'epg-gpg-program))

which gives me this in *Messages* (reformatted slightly):

(theme-value ((use-package "/path/to/gpg2"))
 saved-value ("/path/to/gpg2")
 saved-variable-comment "Customized with use-package epg")

(theme-value ((use-package "/path/to/gpg2"))
 saved-value nil
 saved-variable-comment "Customized with use-package epg"
 standard-value ((funcall #'#[0 "��!�����" [executable-find "gpg2" "gpg"] 2 ""]))
 custom-version "25.1"
 custom-type string
 variable-documentation "..."
 custom-requests nil)

So, use-package seems to set things up fine, but then loading epg apparently tramples the saved-value property?

epg-find-configuration specifically checks the saved-value and customized-value properties for the symbol before falling back to its own lookup logic (which doesn’t find a program on my system), culminating in the “no usable configuration” error.

However, if I instead change my init file to

(custom-set-variables
  '(epg-gpg-program "/path/to/gpg2" nil () "Customized without use-package"))

(use-package epg :init (print (symbol-plist 'epg-gpg-program)))

(print (symbol-plist 'epg-gpg-program))

I get this in *Messages*:

(theme-value ((user "/path/to/gpg2"))
 saved-value ("/path/to/gpg2")
 saved-variable-comment "Customized without use-package")

(theme-value ((user "/path/to/gpg2"))
 saved-value ("/path/to/gpg2")
 saved-variable-comment "Customized without use-package"
 standard-value ((funcall #'#[0 "��!�����" [executable-find "gpg2" "gpg"] 2 ""]))
 custom-version "25.1"
 custom-type string
 variable-documentation "..."
 custom-requests nil)

I.e., everything is correct and works.

Most packages don’t explicitly check the symbol properties, but just use the value directly. In that case, you can’t tell the difference. But the saved-value property seems to be overwritten in all cases when :custom is used, and it causes a problem with a handful of packages.

I don’t know that this is a use-package issue, but I can not track down where this trampling is happening and use-package at least seems to be the place that others are most likely to run into the issue … so here it is.

[sellout]

Ok, so I forgot how powerful emacs debugging is. I added some advice to put and immediately found the culprit.

In custom-declare-variable:

      ;; If there is a value under saved-value that wasn't saved by the user,
      ;; reset it: we used that property to stash the value, but we don't need
      ;; it anymore.
      ;; This can happen given the following:
      ;; 1. The user loaded a theme that had a setting for an unbound
      ;; variable, so we stashed the theme setting under the saved-value
      ;; property in `custom-theme-recalc-variable'.
      ;; 2. Then, Emacs evaluated the defcustom for the option
      ;; (e.g., something required the file where the option is defined).
      ;; If we don't reset it and the user later sets this variable via
      ;; Customize, we might end up saving the theme setting in the custom-file.
      ;; See the test `custom-test-no-saved-value-after-customizing-option'.
      (let ((theme (caar (get symbol 'theme-value))))
        (when (and theme (not (eq theme 'user)) (get symbol 'saved-value))
          (put symbol 'saved-value nil)))

and custom-theme-recalc-variable says

    ;; We used to save VALSPEC under the saved-value property unconditionally,
    ;; but that is a recipe for trouble because we might end up saving session
    ;; customizations if the user loads a theme.  (Bug#21355)
    ;; It's better to only use the saved-value property to stash the value only
    ;; if we really need to stash it (i.e., VARIABLE is void).
    (condition-case nil
        (default-toplevel-value variable) ; See if it doesn't fail.
      (void-variable (when valspec
                       (put variable 'saved-value valspec))))

So, perhaps the issue isn’t that this property is getting clobbered, but that epg is trying to use those properties directly.

Rather, maybe epg should change from

(and symbol (or (get symbol 'saved-value)
                (get symbol 'customized-value)))

to something like

(and symbol
     (or (get symbol 'theme-value)
         (get symbol 'saved-value)
         (get symbol 'customized-value))
     (symbol-value symbol))

i.e., if there is some form of customization, use the symbol-value, rather than trying to deconstruct all the properties?

[sellout]

Although this still doesn’t explain the “CHANGED outside Customize” message. This is because custom-variable-state thinks the variable is 'changed rather than 'themed.

custom-variable-state has a condition

(and (equal comment temp)
     (equal value
            (eval
             (car (custom-variable-theme-value
                   symbol)))))

which, when false, results in 'changed. And it seems like the thing that’s failing with :custom is that (get symbol 'variable-comment) doesn’t match (get symbol 'saved-variable-comment). 'variable-comment isn’t set at all in any of the :custom forms.

It seems like custom-theme-set-variables only sets 'variable-comment if the defcustom form has already been loaded. I removed the conditional from

(and (or now (default-boundp symbol))
     (put symbol 'variable-comment comment))

in that function and now Customize shows me the expected state ('themed) and the comment as I’d expect. I don’t know why that property is conditionalized, though.

[skangas]

There seems to be some subtle error in the interaction between use-package, epg and custom. I believe it would be better if we could move this discussion to the Emacs bug tracker to get to the bottom of this.

Could you please open a bug report using M-x report-emacs-bug RET or by emailing the bug-gnu-emacs mailing list?

Thanks.

[sellout]

@skangas I think there are two distinct issues here. I don’t think either actually involves use-package, but just the “theme” part of custom. I.e., I think I can come up with a minimal reproduction that exonerates use-package, and I’ll submit to Emacs. I’ll close this with references to those issues once I’ve done that.

[skangas]

@sellout Indeed, finding a minimal recipe would be the best way forward. So your plan sounds good to me. Thanks for looking into it.