not clear how to handle config which involves multiple packages

[aspiers]

Hey John :) Firstly, awesome work here. I saw you demo this at emacsconf but only just got round to trying it out. Based on the docs, it seems to cover exactly what I want (the focus on startup speed is especially awesome!), except for one area I am unclear on:

If I have some config which involves integration between multiple packages, how can I use use-package to ensure that config is run only when all those packages are loaded?

For example, guide-key and org-mode, or guide-key and key-chord, or key-chord and any mode which I want key chord bindings for. I'm sure you can easily think of many other combinations :)

If use-package can already do this then I guess this bug is a friendly request to make that more obvious in the README.md, and if it can't, please consider this a feature request :)

[jwiegley]

Interestingly enough, I just finish discussing this issue with someone else.

Right now use-package depends on the linear ordering of declarations within the file. If you want to make sure that a declaration is ignored if another package failed to load, use the :if directive to check for some featurep or some such that indicates whether the other package successfully loaded.

[aspiers]

Ah, but that would prevent one package loading if the other failed to load, and I don't want that...

[jwiegley]

Right, then in that case there is no way to say "Don't load this until X has loaded."

[aspiers]

I see - does closing this issue mean you wouldn't accept pull requests to implement this feature, or just that you don't intend to implement it yourself?

[jwiegley]

I would rather have this feature implemented by an extension package, like use-package-deps or something, rather than in the core. Unless you can show me that (a) it doesn't affect load speed at all, and (b) doesn't introduce excessive levels of complexity in the core code.

[aspiers]

I'd be very surprised if it would affect load speed or introduce complexity. Maybe worth considering what I would want this to look like ... here's an example:

(with-packages (org-mode guide-key)
  (defun guide-key/my-hook-function-for-org-mode ()
    (guide-key/add-local-guide-key-sequence "C-c")
    (guide-key/add-local-guide-key-sequence "C-c C-x")
    (guide-key/add-local-highlight-command-regexp "org-"))
  (add-hook 'org-mode-hook 'guide-key/my-hook-function-for-org-mode))

Then the code would only get run after the :config sections of both packages had been run. Achieving that is presumably pretty easy, but maybe some thought required to make sure it would compile cleanly?

[jwiegley]

I see what you mean, this is different from what I thought you originally meant. We could have a list of "post install" functions which return either nil or t to indicate whether they were able to run, and if they return t then we remove them from the list.

[aspiers]

Oh right :) What did you think I meant?

Yes, that list sounds good. I expect most of the predicates would be of the form (every 'featurep '(org-mode guide-key)), but it might make sense to support arbitrary predicates too.

[aspiers]

I just ran into another two use cases for this:

• (smartrep-define-key global-map "C-S-SPC" ...lots of multiple-cursors bindings...) when both smartrep and multiple-cursors are loaded
• (define-key region-bindings-mode-map "n" 'mc/mark-next-like-this) and similar when both region-bindings-mode and multiple-cursors are loaded

[Silex]

Just wanted to mention https://github.com/edvorg/req-package

[aspiers]

@Silex Thanks a lot, I'll take a look!

[aspiers]

AFAICS this is not really resolved until https://github.com/edvorg/req-package/issues/18 is resolved.

[marcinant]

Personally I solved this by running 'use-package' multiple times on the same package, just with different :if conditions.

[aspiers]

@marcinant Please could you provide an example of that? Even though req-package has supposedly resolved this, I can't find any clear documentation of how it's supposed to work.

[aspiers]

@jwiegley Any chance we could revisit this? req-package has not been updated in 2.5 years, and anyway I get the impression that more "recent" enhancements in use-package such as :after have partially or even fully removed the need for it. But it's still not clear to me how to handle config involving multiple packages. I think there is more need for this than ever, due to the explosion of emacs packages in the last few years which complement each other (e.g. treemacs, ivy, projectile to name a few).

Thanks!

[aspiers]

@marcinant's approach seems flawed, because if the :if condition fails, the config will never get run, but it should get run at the point when all relevant packages are loaded.

I am wondering if a variant of this approach would work, e.g. using my previous example:

(use-package org-mode
  :after guide-key
  :config
  (defun guide-key/my-hook-function-for-org-mode ()
    (guide-key/add-local-guide-key-sequence "C-c")
    (guide-key/add-local-guide-key-sequence "C-c C-x")
    (guide-key/add-local-highlight-command-regexp "org-"))
  (add-hook 'org-mode-hook 'guide-key/my-hook-function-for-org-mode))

But it's not clear whether this would interfere with any other (use-package org-mode ...) stanzas, e.g. preventing any of them from running until guide-key is loaded, which would certainly be undesirable. I still suspect that my previous proposal of a new with-packages macro would be the most user-friendly way to support this.

[aspiers]

Further issues with relying on req-package for this:

• The lack of maintenance means that even though it wraps around req-package, not all features of use-package work (I found this recently, although can't remember off-hand which ones don't). That means users have to mix and match usage of use-package and req-package, which generally feels wrong and makes debugging startup issues with either harder.

• req-package provides this functionality via :require, whereas use-package provides the keyword :requires for something slightly different. That's a confusing situation to be in ;-)

[conao3]

I don't read this whole thread, but use-package is not intended to constitute a nested use-package. This is a consistent view of @jwiegley.

FYI, leaf.el intended to this situation and it have :require keyword.

[aspiers]

Thanks for the reply.

I don't read this whole thread, but use-package is not intended to constitute a nested use-package. This is a consistent view of @jwiegley.

I'm not sure what exactly you mean by "nested", but I don't want to nest anything. I just want to be able to specify config which is automatically activated when a given list of packages are fully loaded.

However I have done some more experiments and I think I have figured it out. It seems that the trick is to include :defer t for the use-package stanza with the combined config. Here is an example, with the combined config in the middle of the three use-package calls:

(package-initialize)

(require 'use-package)
(setq use-package-verbose 'debug)

(use-package org
  :config
  (message ":config for just org")
  :commands org-mode)

(use-package org
  :after ivy
  :defer t
  :config
  (message ":config for org+ivy"))

(use-package ivy
  :config
  (message ":config for just ivy")
  :commands ivy-mode)

(message "Running (ivy-mode)")
(ivy-mode)
(message "------------------------------------")
(message "Running (org-mode)")
(org-mode)

If I run this via emacs -Q --batch -l ~/.emacs.d/test/combined-config.el, I see:

Running (ivy-mode)
Configuring package ivy...
   :config for just ivy
Configuring package ivy...done
------------------------------------
Running (org-mode)
Configuring package org...
   :config for just org
Configuring package org...done
Configuring package org...
   :config for org+ivy
Configuring package org...done

This is the exact desired behaviour - nothing is loaded until it is asked for, and the combined config is only triggered after both packages have loaded.

There is one imperfection: if I swap the order in which ivy-mode and org-mode are called, I see:

Running (org-mode)
Configuring package org...
   :config for just org
Configuring package org...done
------------------------------------
Running (ivy-mode)
Configuring package org...
   :config for org+ivy
Configuring package org...done
Configuring package ivy...
   :config for just ivy
Configuring package ivy...done

In the above, I would expect "just ivy" to appear before "org+ivy", not after. Another experiment revealed that this order is due to the org+ivy use-package being declared before the ivy one; swapping them around fixes the order, but this is not always an option when config is spread over multiple init files.

FYI, leaf.el intended to this situation and it have :require keyword.

That's interesting, and it looks like an impressive piece of work. It would be great if the README.md gave a comprehensive explanation of why you decided to write something from scratch rather than just working with the rest of the community to improve use-package. The current two-sentence explanation is not enough to understand this.

Anyway, given my discoveries, there are two remaining tasks to consider:

1. Document this approach in use-package's README.md.
2. Maybe add a (defmacro with-packages ...) which expands to (use-package a :after b :defer t ...), although I'm not sure how this would work with supporting things like :after (:any (:all foo bar) (:all baz quux)).

[conao3]

Oops, I should be looking at everything in the thread. You want to have fine control over the configuration loading order.

I might be able to solve this with leaf, but it's too off topic. Let's talk more about it in the leaf issue tracker. Anyway, there is a package similar to use-package, like req-package I just wanted to let you know that.

[jwiegley]

@aspiers I'm very interested in fixing the "order of declaration" problem, because using :after should not introduce a dependency on the order in which they're encountered. Do you have perhaps a small test that fails which shouldn't?

[aspiers]

Hi @jwiegley, thanks for the reply. Yes, if you use the above test case but swap the (ivy-mode) call with (org-mode), you'll see the issue I describe.

[aspiers]

To be clear, (use-package org :after ivy...) config is executed before (use-package ivy) config is if and only if its declaration is encountered before ivy's.

[conao3]

This problem is easily solved by a nested use-package. However, as I said before, this usage is not recommended for use-package. Also, since the use-package automatically expands the require, It should be noted that the :no-require keyword is required. Since your script couldn't run in a single file, I need to add the :ensure keyword.

;; ~/.debug.emacs.d/use-package/init.el

;; you can run like 'emacs -q -l ~/.debug.emacs.d/{{pkg}}/init.el'

(prog1 "prepare use-package"
  (prog1 "package"
    (custom-set-variables
     '(package-archives '(("org"   . "https://orgmode.org/elpa/")
                          ("melpa" . "https://melpa.org/packages/")
                          ("gnu"   . "https://elpa.gnu.org/packages/"))))
    (package-initialize))

  (prog1 "use-package"
    (unless (package-installed-p 'use-package)
      (package-refresh-contents)
      (package-install 'use-package))))

(require 'use-package)
(setq use-package-verbose 'debug)

(use-package org
  :defer t
  :config
  (message ":config for just org")
  (use-package *org-ivy-integration
    :no-require t
    :after ivy
    :config
    (message ":config for org+ivy")))

(use-package ivy
  :ensure t
  :defer t
  :config
  (message ":config for just ivy"))

(message "Running (ivy-mode)")
(ivy-mode)
(message "------------------------------------")
(message "Running (org-mode)")
(org-mode)

;; (message "Running (org-mode)")
;; (org-mode)
;; (message "------------------------------------")
;; (message "Running (ivy-mode)")
;; (ivy-mode)

Yes, two of the invocation orders produce the your intended result.

$ emacs -Q --batch -l ~/.debug.emacs.d/use-package/init.el
Running (ivy-mode)
Configuring package ivy...
:config for just ivy
Configuring package ivy...done
------------------------------------
Running (org-mode)
Configuring package org...
:config for just org
Configuring package *org-ivy-integration...
:config for org+ivy
Configuring package *org-ivy-integration...done
Configuring package org...done

$ emacs -Q --batch -l ~/.debug.emacs.d/use-package/init.el
Running (org-mode)
Configuring package org...
:config for just org
Configuring package org...done
------------------------------------
Running (ivy-mode)
Configuring package ivy...
:config for just ivy
Configuring package ivy...done
Configuring package *org-ivy-integration...
:config for org+ivy
Configuring package *org-ivy-integration...done

You can use leaf to produce the same result, but omit some keywords. FYI.

;; ~/.debug.emacs.d/leaf-use-package/init.el

;; you can run like 'emacs -q -l ~/.debug.emacs.d/{{pkg}}/init.el'
(when load-file-name
  (setq user-emacs-directory
        (expand-file-name (file-name-directory load-file-name))))

(prog1 "prepare leaf"
  (prog1 "package"
    (custom-set-variables
     '(package-archives '(("org"   . "https://orgmode.org/elpa/")
                          ("melpa" . "https://melpa.org/packages/")
                          ("gnu"   . "https://elpa.gnu.org/packages/"))))
    (package-initialize))

  (prog1 "leaf"
    (unless (package-installed-p 'leaf)
      (package-refresh-contents)
      (package-install 'leaf))))

(leaf org
  :config
  (message ":config for just org")
  (leaf *org-ivy-integration
    :after ivy
    :config
    (message ":config for org+ivy")))

(leaf ivy
  :ensure t
  :config
  (message ":config for just ivy"))

(message "Running (ivy-mode)")
(ivy-mode)
(message "------------------------------------")
(message "Running (org-mode)")
(org-mode)

;; (message "Running (org-mode)")
;; (org-mode)
;; (message "------------------------------------")
;; (message "Running (ivy-mode)")
;; (ivy-mode)

[aspiers]

Very interesting - thanks! IMHO it should be possible to do this without nesting, or relying on any particular ordering of the use-package declarations. Your :no-require t approach gave me an idea to try something like:

(use-package org
  :config
  (message "   :config for just org")
  :commands org-mode)

(use-package *org+ivy
  :no-require t
  :after (org ivy)
  :config
  (message "   :config for org+ivy"))

(use-package ivy
  :config
  (message "   :config for just ivy")
  :commands ivy-mode)

If that had worked, then it would probably be quite easy to write a with-packages macro to generate the combined declaration.

But unfortunately the :config for *org+ivy still gets run before the :config for ivy. Adding :defer t to it simply prevented *org+ivy from running at all.

So I wonder if this is some kind of subtle bug in when use-package chooses to run :config for packages with an :after clause. @jwiegley perhaps this line of thought might trigger a light-bulb moment for you?

[aspiers]

Here is a draft of the kind of macro I envision:

(defmacro with-packages (when &rest args)
  "Allow declaration of `use-package' config which is only
triggered when a required set of packages are loaded.  The
required set is defined by the `when' argument, whose value is
exactly the same format as with the `:after' argument to
`use-package'.

The `args' are passed straight to `use-package' for use as normal.

Example usage:

  (with-packages (org counsel)
    :bind (:keymap org-mode \"C-c C-j\" . counsel-org-goto))"
  (let ((pseudo-pkg-name
         (format "*with-packages/%s"
                 (replace-regexp-in-string
                  "(\\(.+\\))" "\\1"
                  (s-replace-all '((" " . "-") (":any" . "any") (":all" . "all"))
                                 (prin1-to-string when))))))
    `(use-package ,pseudo-pkg-name
       :no-require t
       :ensure nil
       :after when
       ,@args)))

There are some issues with this - for example it can declare packages with names like *with-packages/any-(all-a-b)-(all-c-d) which I'm not sure is ideal. Also I guess it should avoid a dependency on s.el, but that's easily fixed.

[conao3]

use-package is itself an advanced DSL, so build more DSL on top of it I disagree with that. If you want to do it, As @jwiegley says, you'll want to make another package. I guess.

[aspiers]

I just realised that this is also needed to get nice indentation:

(put 'with-packages 'lisp-indent-function 'defun)

[aspiers]

OK, my with-packages macro is approaching something vaguely functional. If anyone wants to try it, it's here:

https://github.com/aspiers/emacs/blob/master/.emacs.d/lib/with-packages.el

[jwiegley]

Very cool @aspiers. Btw, here is how I would do that:

   (use-package org-counsel
     :no-require t
     :after (:and org counsel)
     :bind (:map org-mode-map
                 (("C-c C-j" . counsel-org-goto))))

[aspiers]

That's almost exactly what my with-packages macro will expand to ;-) The key differences:

• The package name is prefixed with *with-packages/
• It also includes :defer nil :ensure nil :straight nil

and of course the syntactic sugar ensures a consistent approach, prevents mistakes, and promotes legibility.

[skangas]

Would this issue be resolved by adding something like what John suggests in https://github.com/jwiegley/use-package/issues/71#issuecomment-680956294 to the documentation? Or is there anything else that needs to be done here?

[aspiers]

While adding something to the docs is definitely better than nothing, IMHO it's definitely sub-optimal for users to have to write code achieve this. As I said, the syntactic sugar ensures a consistent approach, prevents mistakes, and promotes legibility.

So I think the ideal solution would be to provide the syntactic sugar as part of use-package.

The second best option would be to provide with-packages as a separate package extending use-package, but that is a bit less ideal given that it relies on the use-package API, and could break if the API changed.

[jwiegley]

In the example that I mentioned above, perhaps I don't want :defer nil, for example. Is there a reason we need more that what this provides? Perhaps I've not fully understood your use case...

 (use-package org-counsel
     :no-require t
     :after (:and org counsel)
     :bind (:map org-mode-map
                 (("C-c C-j" . counsel-org-goto))))

[aspiers]

@jwiegley commented on January 11, 2023 6:18 AM:

In the example that I mentioned above, perhaps I don't want :defer nil, for example.

Well I can't remember why I added that in the macro, so maybe it works fine without. That's not the main point of the macro anyway.

Is there a reason we need more that what this provides? Perhaps I've not fully understood your use case...

 (use-package org-counsel
     :no-require t
     :after (:and org counsel)
     :bind (:map org-mode-map
                 (("C-c C-j" . counsel-org-goto))))

As I mentioned above:

• The macro creates a package name is prefixed with *with-packages/, avoiding accidentally collisions (e.g. in your example, imagine if there actually was a package called org-counsel - and indeed this is not at all unlikely; consult-flycheck is one of several examples I've come across).
• The syntactic sugar provides several benefits:
  • It hides complexity and eliminates the need for specific domain-knowledge. In your example, it saves the user from needing to know that :no-require t is required. In mine, :straight nil is another implementation detail.
  • It ensures a consistent approach and prevents mistakes.
  • It promotes legibility, because you can immediately see the difference between a use-package and a with-packages.

[jwiegley]

I don't think the approach you're taking offers enough clarity for users. Since it's a new a macro, you can certainly offer it as an extension to use-package on your own, but I am closing this as an issue in use-package itself that needs addressing here.

[aspiers]

OK I'll try to find time to package myself. Do you have any suggestions how it could offer greater clarity?

[jwiegley]

Just that it should be very clear when and why a person would use this additional macro, rather than the current :after support. I'm using :after in many places, so I'm not clear yet on why I would need or want to introduce the use of a new macro, for example.

[aspiers]

The problem is that the current documentation doesn't make it clear how to achieve this with :after and :no-require either.

IIUC, in your example above, org-counsel is not a real package - it's a "virtual package" used to pull this trick in combination with :no-require t. This requires some specific bits of undocumented domain knowledge:

1. It's possible to write (use-package arbitrary-symbol-not-real-package ...)
2. It's possible to use :no-require t in combination with that in order to achieve some configuration which only activates after a combination of packages loads, without that configuration actually triggering the loading of those packages specified in :after.

The closest the existing docs gets to this is the example:

(use-package ivy-hydra
  :after (ivy hydra))

but that's substantially different, because ivy-hydra is a real package, and therefore :no-require t is not appropriate.

In my original request above, I wrote:

If use-package can already do this then I guess this bug is a friendly request to make that more obvious in the README.md, and if it can't, please consider this a feature request :)

Clearly I've failed to persuade you via the reasons stated above that a macro is the cleanest, most user-friendly way of covering this use case. But with the status quo, I think the only way the vast majority of users would be able to figure out the trick is to go hunting, stumble upon this GitHub issue, and find your trick buried inside it.

So the obvious alternative would be to enhance the README.md to explain to users how to achieve it themselves via the virtual package name trick. Maybe I can find some time to submit a PR for that.

[jwiegley]

@aspiers What if we introduced a new keyword to make the idiom and intent clearer, like :virtual-package t? Then the semantics could vary slightly to take this into account, and avoid needing a separate macro?

[aspiers]

That sounds like a pretty nice idea! So :virtual-package t would imply :no-require t? And it would prevent any possibility of clashing with a real package with the same name? If so, I think that would go a long way towards solving the issues the macro solves.

I wonder if it would also make sense to include any of :defer nil :ensure nil :straight nil - although if some/all of those are too prescriptive to be automatically activated by :virtual-package t then people like me could still use a third-party macro which wraps around (use-package vpkg :virtual-package t ...) to achieve the desired result.

[jwiegley]

I think we could do this in a modular way: Add a virtual-package keyword that relies on a use-package-virtual-package-attributes to define the default set of what happens, keyword-wise, to any such package definition. It would also have some additional semantics, perhaps, like raising an error if a physical package of that same name does in fact exist?

[aspiers]

That makes sense. So then we'd end up with usage like this, right?

(use-package org-counsel
     :virtual-package t
     :after (:and org counsel)
     :bind (:map org-mode-map
                 (("C-c C-j" . counsel-org-goto))))

It's definitely better than the status quo, so I'd be happy if that was included. Although TBH I still don't see the advantage of that vs. a simple macro:

(with-packages (org counsel)
     :bind (:map org-mode-map
                 (("C-c C-j" . counsel-org-goto))))

which automatically constructs the virtual package name to prevent collisions with real packages, and clearly differentiates multi-package setup from the setup of a single package via use-package.

Then the semantics could vary slightly to take this into account, and avoid needing a separate macro?

Maybe I'm missing some reason why it's worth avoiding needing a separate macro?

[jwiegley]

Right now use-package is a one macro offering. Adding new macros changes that. So unless there's a truly compelling reason, describing a package as "virtual" seems to fit better.