use-package and package-autoremove don't cooperate

[DSMasterson]

package-autoremove reports a number of packages that can be removed that are really dependencies of other packages. As it turns out, I believe that the packages were installed by use-package. I think use-package is not properly updating the dependency information that package.el uses.

[a13]

use-package itself doesn't install packages

[DSMasterson]

Question/Bug still stands...

Does use-package call package.el functions in such a way that package-autoremove will say that the package should be removed in some scenario when it really shouldn't be? I have a guess about this, but I'm not familiar with use-package or package.el internals, so I'll leave it to the experts. I have seen package-autoremove attempt to remove packages that should be dependencies according to use-package (via :after), but were not installed via package-list-packages.

[a13]

should be dependencies according to use-package (via :after)

:after doesn't have anything to do with dependencies, the only thing it does is loading one package after another

I believe use-package doesn't resolve package dependencies at all (since it's not a PM) and leaves it to real package managers:

package.el - for :ensure, quelpa and straight for the corresponding keywords

[a13]

You can always see what's going on under the hood of use-package (or any other elisp macro) by calling M-x pp-macroexpand-last-sexp after the form.

An example:

(use-package eshell-prompt-extras
  :ensure t
  :after (eshell esh-opt))

becomes

(progn
  (use-package-ensure-elpa 'eshell-prompt-extras
                           '(t)
                           'nil)
  (defvar use-package--warning213
    #'(lambda
        (keyword err)
        (let
            ((msg
              (format "%s/%s: %s" 'eshell-prompt-extras keyword
                      (error-message-string err))))
          (display-warning 'use-package msg :error))))
  (condition-case-unless-debug err
      (eval-after-load 'esh-opt
        '(eval-after-load 'eshell
           '(if
                (not
                 (require 'eshell-prompt-extras nil t))
                (display-warning 'use-package
                                 (format "Cannot load %s" 'eshell-prompt-extras)
                                 :error))))
    (error
     (funcall use-package--warning213 :catch err))))

[a13]

As for package.el, dependencies are described inside Package-Requires header in the package file itself

[DSMasterson]

should be dependencies according to use-package (via :after)

:after doesn't have anything to do with dependencies, the only thing it does is loading one package after another

I believe use-package doesn't resolve package dependencies at all (since it's not a PM) and leaves it to real package managers:

package.el - for :ensure, quelpa and straight for the corresponding keywords

Hmm. I thought that was dependencies -- ie. package "depends" another package as defined by ":after", so use-package sees if the ":after" package is loaded (which causes it to be loaded via ":ensure").

I was trying to use this to build a .emacs that is self-contained. That is, I go to a new "job", plop down my .emacs and it would take care of loading all the packages I need (with, perhaps, a variable setting for the type of "job"). I get the most up-to-date packages and I don't have to carry around a lot of files.

[DSMasterson]

As for package.el, dependencies are described inside Package-Requires header in the package file itself

Are Package-Requires headers used by package-autoremove in determining what could be removed? Are the headers complete enough to depend on?

[a13]

Hmm. I thought that was dependencies -- ie. package "depends" another package as defined by ":after", so use-package sees if the ":after" package is loaded (which causes it to be loaded via ":ensure").

you may call them loading dependencies, not package ones (you can load one package after internal ones, so there's nothing to install at all), so if you want them to be installed you have to ensure them explicitly

I was trying to use this to build a .emacs that is self-contained.

I have an example in my profile, just one self-containing bootstrapping file (okay, two files, the source in org format and the resulting init.el)

[a13]

As for package.el, dependencies are described inside Package-Requires header in the package file itself

Are Package-Requires headers used by package-autoremove in determining what could be removed? Are the headers complete enough to depend on?

I suppose they are

[fixmaker]

I am seeing the same problem. Given the following code in my init.el:

(use-package magit
  :ensure t
  :bind (("C-x g" . magit-status)))

use-package downloads and installs the package as expected. However when I call M-x list-packages then I see the magit package listed as a dependency, rather than as an installed package.

I think the reason is that the package name is not being added to the package-selected-packages list, but I can't work out why. Looking at use-package-ensure.el it seems that it calls (package-install package) which (according to the docs) should add the package to package-selected-packages.

[DSMasterson]

I am seeing the same problem. [...]ges use-package downloads and installs the package as expected. However when I call M-x list-packages then I see the magit package listed as a dependency, rather than as an installed package.

I think the reason is that the package name is not being added to the package-selected-packages list, but I can't work out why. Looking at use-package-ensure.el it seems that it calls (package-install package) which (according to the docs) should add the package to package-selected-packages.

That's where I think it falls down. package-install is not adding it to package-selected-packages, so that it can be used to install new packages or dependent packages -- it leaves it to the wrapper around package-install to decide. I don't believe that package-autoremove is building a full dependency graph for all packages on the system because I don't think Package-Requires is well maintained.

Therefore, use-package is not cooperating with package.el.

[a13]

@fixmaker can't reproduce with clean (which contains use-package installation only) config and elpa

  magit              20201019.1115 installed             A Git porcelain inside Emacs.

do you use any packages which require magit as a dependency?

[DSMasterson]

I use magit and forge. I think I installed them with package-list-packages.

(setq use-package-always-ensure t)

(use-package magit :bind ("C-x g" . magit) :config (setq magit-view-git-manual-method 'woman) :init (setq auth-sources '("~/.authinfo")) )

(use-package forge :after magit )

[Stebalien]

In case anyone here is looking for a slightly more reliable auto-remove solution, I've advised the ensure and quelpa handlers to record installed packages. I define a new use-package-selected-packages instead of using the one provided by package to ensure that only packages installed by use-package are tracked.

(defvar use-package-selected-packages '(use-package)
  "Packages pulled in by use-package.")
(defun use-package-autoremove ()
  "Autoremove packages not used by use-package."
  (interactive)
  (let ((package-selected-packages use-package-selected-packages))
    (package-autoremove)))

(eval-and-compile
  (define-advice use-package-handler/:ensure (:around (fn name-symbol keyword args rest state) select)
    (let ((items (funcall fn name-symbol keyword args rest state)))
      (dolist (ensure args items)
        (let ((package
               (or (and (eq ensure t) (use-package-as-symbol name-symbol))
                   ensure)))
          (when package
            (when (consp package)
              (setq package (car package)))
            (push `(add-to-list 'use-package-selected-packages ',package) items))))))
  (define-advice use-package-handler/:quelpa (:around (fn name-symbol keyword args rest state) select)
    (let ((package (pcase (car args)
                     ((pred symbolp) (car args))
                     ((pred listp) (car (car args))))))
      (cons `(add-to-list 'use-package-selected-packages ',package)
            (funcall fn name-symbol keyword args rest state)))))

[azuk]

I am seeing the same problem. [...]ges use-package downloads and installs the package as expected. However when I call M-x list-packages then I see the magit package listed as a dependency, rather than as an installed package. I think the reason is that the package name is not being added to the package-selected-packages list, but I can't work out why. Looking at use-package-ensure.el it seems that it calls (package-install package) which (according to the docs) should add the package to package-selected-packages.

That's where I think it falls down. package-install is not adding it to package-selected-packages, so that it can be used to install new packages or dependent packages -- it leaves it to the wrapper around package-install to decide. I don't believe that package-autoremove is building a full dependency graph for all packages on the system because I don't think Package-Requires is well maintained.

Therefore, use-package is not cooperating with package.el.

As earlier commenter said, package-install should add its argument to package-selected-packages, unless explicitly told not to, as described in its documentation, and seems to do so when I tested it.

That said, I've also suffered about the problem of package-autoremove suggesting that I could remove packages that I've installed with use-package, and I believe some that I've directly installed with package-install.

We must note that use-package does not actually call package-install if package-installed-p returns non-nil for the package being "ensured". It's possible that the package is installed but not in package-selected-packages, even though it should be. One cause for that could be that it was first installed before the issue #327 was fixed. Or it was installed using an incompatible package.el as suggested in #353. Also, because package-selected-packages is saved to wherever customizations are saved, you might copy your installed packages (.emacs.d directory) to another machine but not your customizations, if they are in a separate file outside of .emacs.d and you treat them as "local" as I do.

Hard to say what was the original cause for my problem but the situation remained broken until I explicitly called package-install for all packages so that package-selected-packages got updated to match my use-package usage.

[felker]

@azuk thanks for your latest post; it helped me with my own confusion with package-autoremove and use-package's ensure. This is another useful post that seems to backup your conclusions: https://www.reddit.com/r/emacs/comments/np6ey4/how_packageel_works_with_use_package/

[michaelmhoffman]

Function package-install calls function package--save-selected-packages, which only adds to package-selected-packages when variable after-init-time is non-nil. Otherwise, it adds package--save-selected-packages to after-init-hook.

I suspect that somehow package--save-selected-packages is falling through the cracks during some typical invocations of use-package. Manually executing a use-package form for an uninstalled package seems to add it to package-selected-packages just fine, when it wasn't otherwise added. For example, I often run use-package through running batch-byte-compile from the command-line, which probably has nil after-init-time and after-init-hook never gets run.

[michaelmhoffman]

The package--save-seleted-packages code in question was added in emacs-mirror/emacs@d0a5162fd825acbbd863e61099e1fa1ce5975773 to fix bug 20855.

It seems like package--save-seleted-packages could be changed to only use the after-init-hook path in fewer conditions—such as when package-selected-packages doesn't have the saved-value property indicating that custom has already loaded it. If noninteractive is non-nil should probably yield a warning too. But all that would be for Emacs and not for use-package.

[quite]

I'm also running in to this now. Sounds like this could be something that ought to be fixed up in emacs then, did you file a bug there @michaelmhoffman ?