[Feature]: optionally install packages synchronously

[Luis-Henriquez-Perez]

Feature Description

Either a certain parameter of a function or macro, or a variable (probably the former would be preferable) that forces elpaca to install packages synchronously.

when asynchronous installation is desirable

I understand that elpaca is specifically designed to work asynchronously so a synchronous install would be an "anti-feature".

Sometimes in which an asynchronous install of packages is preferable. For me this is nice when installing a package during an active emacs session. You can use elpaca-try to install one or more packages, and do other things while waiting for them to be installed.

recommended solution: put your init in elpaca-after-init-hook

The solution suggested for why did my init file execute out of order is to move all of your init file (or at least the external package dependant parts of it) into elpaca-after-init-hook. This way this code will be run only after package installation as ended.

practical problem to asynchronous installation : missing timing

Even if you follow the recommended solution, you still miss potentially the entire startup process when installing a sufficiently large number of packages. So any code you have involving after-init-hook, before-init-hook, frame-notice-user-settings, .etc has to be revised and likely modified so that it still works even if it missed timing.

stylistic and idealogical problems with suggested solution

In my opinion, requiring to move all their init file contents into a hook to accommodate a single emacs package is unreasonable. A package should require installation, configuration and installation of any dependencies it needs, but it shouldn't entail a restructuring of your entire config solely to meet its own needs.

installing packages asynchronously at startup does not benefit many users

I suspect many Emacs users like me have a config file that heavily relies on external packages--so much so that not having those packages would make emacs unusable for them. For these users, an asynchronous package installation during startup does not benefit them as much because even if emacs is unblocked, it would be too inconvenient to use until their main packages are installed.

others who have mentioned this

I am not the only one who has been concerned about this. I assert that enough users have expressed interest in a synchronous installation for it to be added as an alternative.

https://github.com/progfolio/elpaca/issues/29#issuecomment-1323506198 https://github.com/progfolio/elpaca/issues/29#issuecomment-1345689506 https://github.com/progfolio/elpaca/issues/7#issuecomment-1300662283 https://github.com/progfolio/elpaca/issues/7#issuecomment-1253367568

Confirmation

• [X] The feature I'm proposing does not already exist in Elpaca

[progfolio]

So any code you have involving after-init-hook, before-init-hook, frame-notice-user-settings, .etc has to be revised and likely modified so that it still works even if it missed timing.

Elpaca could defer such hooks/functions and run them after all its queues are processed. Of course, there will be corner cases if the process is deferred, but I don't think this will be a major issue for most use cases. If you have specific examples of usage for these, especially where Elpaca is breaking what you're trying to do, I would appreciate them. It will help me evaluate the usefulness of the above strategy.

In my opinion, requiring to move all their init file contents into a hook to accommodate a single emacs package is unreasonable. A package should require installation, configuration and installation of any dependencies it needs, but it shouldn't entail a restructuring of your entire config solely to meet its own needs.

Using use-package requires restructuring one's init file. Using striaght.el requires some restructuring as well. I don't think it's unreasonable to require some restructuring. I also think it's best to minimize the amount of restructuring needed. Elpaca could be improved with that in mind, but it will likely always require some restructuring.

installing packages asynchronously at startup does not benefit many users

This I disagree with.

an asynchronous package installation during startup does not benefit them as much because even if emacs is unblocked, it would be too inconvenient to use until their main packages are installed.

The point of asynchoronous installation isn't so the user can get a bunch of work done while their packages install. The main reasons are:

It allows one to better observe the installation process. If you've ever had a package fail to install or a spotty network connection with straight.el, it's hard (or impoossible) to know what's going on or why because Emacs is blocked. Elpaca does its best to log everything as it's happening. It has considerably improved insight into the installation/building process of packages. One even has a chance at recovering from a failed/blocked package installation more easily.

It dramatically decreases installation time, especially when initially installing all of one's packages. This would not be possible if packages were installed synchronously. This may not seem like a big deal until you consider all the wasted time for every user's install (which has to be restarted from scratch if anything goes wrong).

I assert that enough users have expressed interest in a synchronous installation for it to be added as an alternative.

Regarding the comments you've linked to, I've asked all of those users for more details, examples, etc. I have not received specifics which are necessary to determine the severity of the problems as well as possible solutions to the problems.

I'll keep thinking on possible solutions. I appreciate your continued input. Thank you.

[Luis-Henriquez-Perez]

This I disagree with

You're right. When I said this I didn't mean that there were not also many users that did benefit from installing asynchronously. But even still the points you make are compelling.

It allows one to better observe the installation process. If you've ever had a package fail to install or a spotty network connection with straight.el, it's hard (or impoossible) to know what's going on or why because Emacs is blocked. Elpaca does its best to log everything as it's happening. It has considerably improved insight into the installation/building process of packages. One even has a chance at recovering from a failed/blocked package installation more easily.

I stand corrected. Admittedly, I did not think about these points that you've mentioned. And they are compelling points. Maybe they could be mentioned in the readme--like the reasoning behind the design choice of being asynchronous versus synchronous, the benefits and drawbacks of such a choice.

Using use-package requires restructuring one's init file.

If you install use-package, it's because you already consciously buy into the "use-package way" of structuring configuration. I would not consider it a prerequisite of the package. It's the main feature of the package. In other words, I don't think people restructure their config to use use-package. Rather, they use use-package to restructure their config (because they like the DSL).

Using straight.el requires some restructuring as well.

With straight you can keep your init file the same, just prepending the straight bootstrap code and straight-use-package calls before it.

If "restructuring" were just prepending code it would be fine. It's the skipping of the startup process that's the most concerning to me.

Then again, the more I think about it the more I start to think that problem is that we're trying to treat an emacs session where all of our packages have to be installed the same as one where all of our packages are already installed. Maybe it's best to just accept that these cases are not the same and to separate installation and configuration.

If we think of the the installing as separate, then the issue of the startup process is irrelevant. I can start one emacs session whose sole purpose is to install packages and prompt me to restart emacs when it's done.

Curious, how does a script work with asynchronous installation? My concern is that a script that bootstraps elpaca and installs all your packages from the recipe file won't work because it will finish before packages finish installing since it's not synchronous.

It dramatically decreases installation time, especially when initially installing all of one's packages. This would not be possible if packages were installed synchronously.

I am curious about the costs and benefits as I am not too familiar about asynchronous processing. The same work is being done either way right? So in terms of system resources, are synchronous and asynchronous processing the same?

This may not seem like a big deal until you consider all the wasted time for every user's install (which has to be restarted from scratch if anything goes wrong).

Good point.

I have not received specifics which are necessary to determine the severity of the problems as well as possible solutions to the problems.

I have at least one example in mind, but within a couple of days I will experiment with re-installing my packages to provide a better picture.

[progfolio]

Maybe they could be mentioned in the readme--like the reasoning behind the design choice of being asynchronous versus synchronous, the benefits and drawbacks of such a choice.

I agree this would be useful. I'm aiming to keep the README brief (in comparison to straight.el), but I should have a section on the motivation/goals of the project. I'll add that to my TODO list.

It's the skipping of the startup process that's the most concerning to me.

I'll see if I can come up with a solution which simply defers the steps of the startup process and runs them after Elpaca has processed its queues. That should cover most cases. I can't promise 100% compatibility, but it may be an improvement over what Elpaca does now.

Maybe it's best to just accept that these cases [installation vs loading] are not the same and to separate installation and configuration.

Most of what Elpaca does is build a data-structure associated with a package, then execute a series of "build" steps. A pseudo-code outline of the process:

(queue-packages)
(cl-loop with completed
         for package in queued-packages do
         (cl-loop with steps = (if (built? package) '(activate) '(install build activate))
                  for step in steps (funcall step package)
                  finally do (push package completed))
         (mapc #'load-package completed))

Note the example is facile for simplicity's sake, but it does show how the two cases are similar. They only differ in the "steps" that need to be done. This approach also handles a case where a package has been deleted and needs to be re-installed, or a new package has been added to the init file and needs to be installed, etc, without the user having to manually remember to run a separate installation script.

Curious, how does a script work with asynchronous installation? My concern is that a script that bootstraps elpaca and installs all your packages from the recipe file won't work because it will finish before packages finish installing since it's not synchronous.

You're correct that if --batch were used, the Emacs process would exit prior to Elpaca's subprocesses finishing. However, Emacs could be started as a daemon, install everything asynchronously, and then kill the daemon. e.g.

emacs -q --daemon="install" --eval '(load-file "install.el")'

with install.el containing:

;;bootstrap snippet
(elpaca 'example)
(elpaca-process-queues)

I just experimented with something similar and everything was installed. One also has the benefit of connecting via emacsclient -s "install" if anything goes wrong to investigate and possibly correct things.

To your earlier point, it looks like -q prevents us from having a way of automatically killing the Emacs daemon via elpaca-after-init-hook. Such a script would also want to advise the logging functionality so it was printed to stdout. I'm sure both of these rough edges could be worked around, though.

I have at least one example in mind, but within a couple of days I will experiment with re-installing my packages to provide a better picture.

Thank you. Concrete examples are appreciated. They give me something to work off of and test against.

[Luis-Henriquez-Perez]

I'm aiming to keep the README brief (in comparison to straight.el), but I should have a section on the motivation/goals of the project.

As the README is something I've been meaning to talk about I created #43 to discuss it.

I'll see if I can come up with a solution which simply defers the steps of the startup process and runs them after Elpaca has processed its queues. That should cover most cases.

I can't help but have a nagging feeling that this will be fragile. It sounds kind of like a hack. It will also mean that should any new startup stuff is added, elpaca will need be updated which isn't ideal (though it's not like that will happen often). That being said, if you come up with a way to do this stably and reliably I don't see a problem.

A pseudo-code outline of the process

Thanks for this outline. Although "facile" as you mentioned it helps to understand the core concept.

This approach also handles a case where a package has been deleted and needs to be re-installed, or a new package has been added to the init file and needs to be installed, etc, without the user having to manually remember to run a separate installation script.

If the startup process can be properly deferred and there are no other problems that stem from that, then I would be fine with this approach.

You're correct that if --batch were used, the Emacs process would exit prior to Elpaca's subprocesses finishing. However, Emacs could be started as a daemon, install everything asynchronously, and then kill the daemon.

This sounds promising.

One also has the benefit of connecting via emacsclient -s "install" if anything goes wrong to investigate and possibly correct things.

I really like the possibility to debug the installation process with emacs. This is a big advantage over a bash script.

Such a script would also want to advise the logging functionality so it was printed to stdout. I'm sure both of these rough edges could be worked around, though.

I'll see if any progress can be made with the startup process determent before delving into this route.

[progfolio]

elpaca-wait should address some of your concerns. https://github.com/progfolio/elpaca/issues/7#issuecomment-1368164508

Adding another (elpaca-wait) at the end of the init file in the above comment (or at the very start of after-init-hook) should allow one to have everything installed/loaded and then resume the normal start up process. Let me know how it goes if you try it out.

[Luis-Henriquez-Perez]

I get an error using git checkout: error: pathspec 'feat/new-elpaca-wait' did not match any file(s) known to git.

I think it's because I need to fetch the remote branches. Manually going to the directory and executing git fetch --all && git checkout feat/elpaca-wait works. However for some reason the following does not.

((zerop (call-process "git" nil buffer t "clone" (plist-get order :repo) repo)))
(default-directory repo)
((zerop (call-process "git" nil buffer t "fetch" "--all")))
((zerop (call-process "git" nil buffer t "checkout" (or (plist-get order :ref) "--"))))

I'm working on figuring it out, maybe using call-process-shell-command instead.

[progfolio]

Sorry, that branch name has been updated. Try "feat/elpaca-wait"

[Luis-Henriquez-Perez]

Sorry, that branch name has been updated. Try "feat/elpaca-wait"

Thanks, that worked for me.

I am using the following snippet. The defvar declarations and when-let form correspond to the comment you linked. Is this the proper use of elpaca-wait? I'm not sure whether I need the after-init-hook to elpaca-process-queues.

----defvar declarations + when-let form----
(require 'elpaca)
(require 'elpaca-autoloads)
(add-hook 'after-init-hook #'elpaca-process-queues)
(elpaca `(,@elpaca-order))
(dolist (recipe (oo-read-recipes)) (eval `(elpaca ,recipe) t))
(elpaca-wait)
(load (expand-file-name "config.el" user-emacs-directory))

Yesterday, running this snippet I get an error saying that the main elisp file for elpaca cannot be found. But today I can't seem to reproduce this error (I was trying to figure out how to screenshot it). Wierd.

Anyway today it worked very well when installing from scratch.

However I have get a void-function error on elpaca-wait when I tried closing emacs and opening it again with all the packages already installed.

[Luis-Henriquez-Perez]

Modifying the init code as such seems to resolve these problems. Though now I should test starting from scratch with this.

(require 'elpaca)
(require 'elpaca-autoloads)
(elpaca `(,@elpaca-order))
(dolist (recipe (oo-read-recipes)) (eval `(elpaca ,recipe) t))
(elpaca-process-queues)
(elpaca-wait)
(load (expand-file-name "config.el" user-emacs-directory))

[progfolio]

Is this the proper use of elpaca-wait? I'm not sure whether I need the after-init-hook to elpaca-process-queues.

elpaca-wait calls elpaca-process-queues internally. You shouldn't need to call it in after-init-hook if you set things up that way.

There is a corner case I haven't implemented a solution for yet. When elpaca-wait is used at the end of an init file, elpaca-after-init-hook is completely bypassed as well as some logic which determines the type of the next queue. I'm still considering the design around this.

Yesterday, running this snippet I get an error saying that the main elisp file for elpaca cannot be found. But today I can't seem to reproduce this error (I was trying to figure out how to screenshot it). Wierd.

Not sure about this one, but feel free to open an issue if it crops up again.

However I have get a void-function error on elpaca-wait when I tried closing emacs and opening it again with all the packages already installed.

Sorry about that. elpaca-wait needs to be autoloaded. I've pushed a fix which includes that. It should work now after a fresh build. Let me know if that's not the case.

[Luis-Henriquez-Perez]

In another trial of installing everything from scratch I seem to have an error installing paredit. I wonder why there are inconsistent issues on startup. If the recipe I used at first worked once, why didn't the same one work again later?

I will try to see if changing the recipe could fix this.

This is when trying to install paredit manually.

And this is the recipe I used for paredit.

[progfolio]

That server offering paredit does not support shallow clones. Adding :depth nil to the recipe wlil request a full repository clone.

[Luis-Henriquez-Perez]

That server offering paredit does not support shallow clones. Adding :depth nil will by request a full repository clone.

Ok.

I tried two more attempts where the installation again worked perfectly. And I additionally added a :depth nil to the recipe of paredit (this was after the two attempts). Hopefully, this will result in the paredit issue not happening again.

Sorry about that. elpaca-wait needs to be autoloaded. I've pushed a fix which includes that. It should work now after a fresh build. Let me know if that's not the case.

Does the commit 647cfb7 on the feat/elpaca-wait branch contain the patch you pushed? I still get the void-function elpaca-wait error on this commit.

[progfolio]

Does the commit 647cfb7 on the feat/elpaca-wait branch contain the patch you pushed? I still get the void-function elpaca-wait error on this commit.

It does. I've confirmed the autoloads are working as expected on my end. I would try it again with a fresh download of Elpaca (you needn't rebuild all the packages, just M-x elpaca-delete elpaca followed by restart-emacs should work).

Try it without altering the bootstrapping snippet other than pointing the :ref for Elpaca to "feat/elpaca-wait":

(defvar elpaca-directory (expand-file-name "elpaca/" user-emacs-directory))
(defvar elpaca-builds-directory (expand-file-name "builds/" elpaca-directory))
(defvar elpaca-order '(elpaca :repo "https://github.com/progfolio/elpaca.git"
                              :ref "feat/elpaca-wait"
                              :build (:not elpaca--activate-package)))
(when-let ((repo  (expand-file-name "repos/elpaca/" elpaca-directory))
           (build (expand-file-name "elpaca/" elpaca-builds-directory))
           (order (cdr elpaca-order))
           ((add-to-list 'load-path (if (file-exists-p build) build repo)))
           ((not (file-exists-p repo))))
  (condition-case-unless-debug err
      (if-let ((buffer (pop-to-buffer-same-window "*elpaca-bootstrap*"))
               ((zerop (call-process "git" nil buffer t "clone"
                                     (plist-get order :repo) repo)))
               (default-directory repo)
               ((zerop (call-process "git" nil buffer t "checkout"
                                     (or (plist-get order :ref) "--")))))
          (progn
            (byte-recompile-directory repo 0 'force)
            (require 'elpaca)
            (and (fboundp 'elpaca-generate-autoloads)
                 (elpaca-generate-autoloads "elpaca" repo))
            (kill-buffer buffer))
        (error "%s" (with-current-buffer buffer (buffer-string))))
    ((error)
     (warn "%s" err)
     (delete-directory repo 'recursive))))
(require 'elpaca-autoloads)
(elpaca `(,@elpaca-order))

(dolist (recipe (oo-read-recipes)) (eval `(elpaca ,recipe) t))
(elpaca-wait)
(load (expand-file-name "config.el" user-emacs-directory))

[Luis-Henriquez-Perez]

I would try it again with a fresh download of Elpaca (you needn't rebuild all the packages, just M-x elpaca-delete elpaca followed by restart-emacs should work).

I tried your exact snippet and I still get the same error. I deleted elpaca. And then I started another emacs instance with:

(start-process "emacs" nil "emacs" "--debug-init")

I don't want to restart emacs because I don't want to break my current config.

When I go to confirm that elpaca is indeed using the correct commit I go inside the elpaca directory and use magit to check.

647cfb7 * feat/elpaca-wait origin/feat/elpaca-wait add elpaca-wait for synchronous queue polling

I want to note that all goes well during an install from scratch, but starting another emacs instance afterwards fails with the void-function error.

I want to note that when I installed elpaca from scratch along with all my packages, everything worked fine. But only after restarting emacs, elpaca-wait was not defined.

And when I instead just tried elpaca-delete followed by restarting emacs I would get the void-function error again.

[progfolio]

Interesting. I would check the contents of ~/.emacs.d/elpaca/repos/elpaca/elpaca-autoloads.el and ensure that is symlinked to under ~/.emacs.d/elpaca/builds/elpaca/elpaca-autoloads.el. It should contain (among other autoload definitions):

(autoload 'elpaca-wait "elpaca" "\
Block until current queues processed.
When quit with \\[keyboard-quit], running sub-processes are not stopped.")

[Luis-Henriquez-Perez]

I will retrace my steps of trying again.

First I set the contents of the init file to your snippet.

Then I call elpaca-delete on elpaca.

Then I open another emacs instance. And get a void-function for elpaca.

Then I try adding a (require 'elpaca) before (require 'elpaca-autoloads).

Then emacs startsup without error.

Then I close the emacs instance and try opening a new emacs instance--changing nothing in my init.el.

[Luis-Henriquez-Perez]

I forgot to mention that before the void-function elpaca I got the main elisp file error again:

[Luis-Henriquez-Perez]

The contents of ~/.config/emacs/elpaca/repos/elpaca/elpaca-autoloads.el. It doesn't look like there's anything in here.

[progfolio]

Thanks for those details. The empty autoloads file has been reported before in #48 . I'm not sure what the cause of this is yet and haven't been able to reproduce it myself.

The other failure is happening during elpaca--dependencies. If you're comfortable using edebug try instrumenting elpaca--dependencies and stepping through it by evaluating the following in your scratch buffer:

(elpaca--dependencies (elpaca-get-queued 'elpaca))

I'm guessing either:

• The build step is being executed at the wrong time
• The elpaca repo is not on disk or that step has incorrectly set default-directory (it should be in elpaca's repo dir)

• The elpaca.el file is not on disk at this point in the build.

  I don't know why any of those would be, but stepping through may help pin it down. If you need any pointers on how to do that or what to look for I can guide you through it.

[Luis-Henriquez-Perez]

If you're comfortable [https://www.gnu.org/software/emacs/manual/html_node/elisp/Using-Edebug.html](using edebug) try instrumenting elpaca--dependencies and stepping through it by evaluating the following in your scratch buffer

I have barely used the edebugger. But I have benefited greatly from this package, so I want to give back. I'll keep you posted when I have a chance to instrument it.

[progfolio]

I want to give back

The testing you are providing is doing just that. Thank you. I had some time to get an Emacs 28 environment set up. I've pushed a potential fix and rebased the "feat/elpaca-wait" branch on top of it. Try out your test case from above and see if you're still getting the same errors.

[Luis-Henriquez-Perez]

My test case gives me the same errors.

Namely

1) void elpaca Then I add (require 'elpaca) before (require 'elpaca-autoloads). 2) void elpaca-log 3) void elpaca-wait

I'm going to start the instrumenting right now.

[progfolio]

I'm going to start the instrumenting right now.

Sounds good. Thank you. I've tried to keep that branch rebased ontop of master. So it should include the fix.

I don't want to restart emacs because I don't want to break my current config.

I've added the elpaca-test macro which should make testing easier. It will install Elpaca in a test environment and launch a new emacs instance using that environment.

[Luis-Henriquez-Perez]

Steps for instrumenting that I take. 1- delete elpaca 2- start a new emacs instance 3- switch to scratch buffer 4- type (elpaca--dependencies (elpaca-get-queued 'elpaca)) 5- call edebug-defun on elpaca-dependencies

Give me a moment I need set up way to take gifs. I think a gif would be much more useful here.

[progfolio]

You may also be able to just M-x toggle-debug-on-error and then evaluate:

(elpaca--dependencies (elpaca-get-queued 'elpaca))

That should hit the debugger. You should be able to copy the text of the backtrace buffer.

[Luis-Henriquez-Perez]

Can I put it in the init file as so? It would make it easier for me than to manually go to the scratch buffer and type it out. Or does this have to be after an error first occurs?

(require 'elpaca)
(elpaca--dependencies (elpaca-get-queued 'elpaca))
(require 'elpaca-autoloads)
(elpaca `(,@elpaca-order))

(dolist (recipe (oo-read-recipes)) (eval `(elpaca ,recipe) t))
(elpaca-wait)
(load (expand-file-name "config.el" user-emacs-directory))

[Luis-Henriquez-Perez]

With this init file:

(require 'elpaca)
(require 'elpaca-autoloads)
(elpaca `(,@elpaca-order))
(message "deps->%S" (elpaca--dependencies (elpaca-get-queued 'elpaca)))

(dolist (recipe (oo-read-recipes)) (eval `(elpaca ,recipe) t))
(message "deps->%S" (elpaca--dependencies (elpaca-get-queued 'elpaca)))
(elpaca-wait)

It was called by the second call to elpaca-dependencies.

And this is the messages buffer.

So the first call returned that the dependency was emacs 27.1.

[progfolio]

Sorry, the screenshots are only partially useful because the text is abbreviated in them. Could you repeat that test with (setq backtrace-line-length nil) before the code which triggers the error? That should get us a full backtrace buffer. It would be useful if you could share the actual text of the whole backtrace buffer rather than an image, too.

[Luis-Henriquez-Perez]

the screenshots are only partially useful because the text is abbreviated in them. Could you repeat that test with (setq backtrace-line-length nil) It would be useful if you could share the actual text of the whole backtrace buffer rather than an image, too.

Understood.

Debugger entered--Lisp error: (wrong-type-argument elpaca< nil)
  signal(wrong-type-argument (elpaca< nil))
  (or (progn (and (memq (car-safe e) cl-struct-elpaca<-tags) t)) (signal 'wrong-type-argument (list 'elpaca< e)))
  (progn (or (progn (and (memq (car-safe e) cl-struct-elpaca<-tags) t)) (signal 'wrong-type-argument (list 'elpaca< e))) (nth 5 e))
  (let* ((default-directory (progn (or (progn (and (memq (car-safe e) cl-struct-elpaca<-tags) t)) (signal 'wrong-type-argument (list 'elpaca< e))) (nth 5 e))) (package (file-name-sans-extension (progn (or (progn (and (memq (car-safe e) cl-struct-elpaca<-tags) t)) (signal 'wrong-type-argument (list 'elpaca< e))) (nth 2 e)))) (name (concat package ".el")) (regexp (concat "^" name "$")) (main (or (plist-get (progn (or (progn (and (memq (car-safe e) cl-struct-elpaca<-tags) t)) (signal 'wrong-type-argument (list 'elpaca< e))) (nth 10 e)) :main) (cl-some #'(lambda (f) (let ((e (expand-file-name f))) (and (file-exists-p e) e))) (list (concat package "-pkg.el") name (concat "./lisp/" name) (concat "./elisp/" name))) (car (directory-files default-directory nil regexp)) (car (elpaca--directory-files-recursively default-directory regexp)) (car (directory-files default-directory nil "\\.el$" 'nosort)) (error "Unable to find main elisp file for %S" package))) (deps (if (not (file-exists-p default-directory)) (error "Package repository not on disk: %S" (progn (or (progn (and (memq (car-safe e) cl-struct-elpaca<-tags) t)) (signal 'wrong-type-argument (list 'elpaca< e))) (nth 10 e))) (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (insert-file-contents-literally main) (goto-char (point-min)) (if (string-suffix-p "-pkg.el" main) (eval (nth 4 (read (current-buffer)))) (let ((case-fold-search t)) (if (re-search-forward "^;+[  ]+\\(Package-Requires\\)[   ]*:[  ]*" nil 'noerror) (progn (let ((deps (list (buffer-substring-no-properties (point) (line-end-position))))) (forward-line 1) (while (looking-at "^;+\\(\11\\|[\11 ]\\{2,\\}\\)\\(.+\\)") (setq deps (cons (match-string-no-properties 2) deps)) (forward-line 1)) (condition-case err (read (mapconcat #'identity (nreverse deps) " ")) ((error) (error "%S Package-Requires error: %S" main err))))))))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))))))) (let* ((--cl-var-- deps) (dep nil) (--cl-var-- nil)) (while (consp --cl-var--) (setq dep (car --cl-var--)) (setq --cl-var-- (cons (if (or (symbolp dep) (null (cdr-safe dep))) (list (elpaca--first dep) "0") dep) --cl-var--)) (setq --cl-var-- (cdr --cl-var--))) (nreverse --cl-var--)))
  elpaca--dependencies(nil)
  (message "deps->%S" (elpaca--dependencies (elpaca-get-queued 'elpaca)))
  eval-buffer(#<buffer  *load*> nil "/home/luis/.config/emacs/init.el" nil t)  ; Reading at buffer position 1782
  load-with-code-conversion("/home/luis/.config/emacs/init.el" "/home/luis/.config/emacs/init.el" t t)
  load("/home/luis/.config/emacs/init" noerror nomessage)
  startup--load-user-init-file(#f(compiled-function () #<bytecode 0xa305acd0865419c>) #f(compiled-function () #<bytecode 0x4d12c0e0e44348b>) t)
  command-line()
  normal-top-level()

[progfolio]

Thanks. The (elpaca--depencencies (elpaca-get-queued 'elpaca)) needs to be after elpaca is queued by

(elpaca `(,@elpaca-order))

[Luis-Henriquez-Perez]

I think the problem was I had two elpaca recipes. I didn't consider that bootstrap file specified a recipe for elpaca as well. One recipe was in my recipes file.

* elpaca
:PROPERTIES:
:ID:       c1a92f04-a31d-4a58-ba40-8bf32dc36f8b
:END:
#+begin_src emacs-lisp
'(elpaca :protocol https :remotes "origin" :inherit t :depth 1 :fetcher github :repo "progfolio/elpaca" :package "elpaca" :files (:defaults) :ref "2805a2e2e8ab28a447e1b885669cbda0ae7359c9")
#+end_src

I think the conflict between the recipes is what caused this error. And it also explains why the second elpaca--dependencies call generated an error. When I removed above recipe from my org file, elpaca started up fine and with no errors. Maybe this was the culprit behind #48 as well?

deps->((emacs "27.1")) [2 times]

I will note that starting emacs again, after elpaca has already been installed the startup time is really slow 74.94 seconds for me. And in theory it shouldn't be because all packages have already been installed. Suprisingly, the startup time when I was actually installing a package--elpaca was 7 times faster, at about 10 seconds.

[progfolio]

I think the conflict between the recipes is what caused this error. And it also explains why the second elpaca--dependencies call generated an error.

Two recipes would gum things up, but there should've been a warning per elpaca--queue:

https://github.com/progfolio/elpaca/blob/5c7bd1132521e6d6a6e0d67455b9ce8283777537/elpaca.el#L740-L741

I'll look into that.

I will note that starting emacs again, after elpaca has already been installed the startup time is really slow 74.94 seconds for me. And in theory it shouldn't be because all packages have already been installed. Suprisingly, the startup time when I was actually installing a package--elpaca was 7 times faster, at about 10 seconds.

Please provide full logs for both scenarios so we can see where the time is being spent. M-x elpaca-log followed by s to search and RET to enter the empty string (which will show everything by default).

[Luis-Henriquez-Perez]

Please provide full logs for both scenarios so we can see where the time is being spent. M-x elpaca-log followed by s to search and RET to enter the empty string (which will show everything by default).

I set about to reproduce this today. While doing this I encountered a different error.

1. I've deleted the elpaca directory
2. I've set up the init file

   ...bootstrap code
   (setq backtrace-line-length nil)
   (require 'elpaca)
   (require 'elpaca-autoloads)
   (elpaca `(,@elpaca-order))
   (dolist (recipe (oo-read-recipes)) (eval `(elpaca ,recipe) t))
   (elpaca-wait)

   

Frustrating is that I do start an emacs instance with --debug-init enabled, but the backtrace isn't showing up. And emacs's is suggesting that I didn't use the debug flag.

The error is cauused by (elpaca-wait) as it appears when I uncomment it and dissapears when I commented.

[progfolio]

Please share your init files (including the recipes file) and I will see if I can reproduce and fix this. Also, is there a *Warnings* buffer? If so, what are the contents?

[Luis-Henriquez-Perez]

I posted a zip file with my init files in this comment.

Also I checked further. Turns out the buffer that talks about number-or-marker-p error is the warnings buffer. And the backtrace buffer is empty for some reason.

The messages buffer (shown below) hints that the error comes form the function elpaca--ibs.

Let me know if there is anything else I can do or provide to help.

[progfolio]

Thanks for testing this more. The void function error for elpaca-log should not happen if the autoloads are properly loaded. Is there a non-empty elpaca-autoloads.el file in builds/elpaca/ and/or repos/elpaca/? If so, I would move (require 'elpaca-autoloads) prior to (require 'elpaca) in your modified bootstrap code. See if that helps.

[Luis-Henriquez-Perez]

Moving the (require 'elpaca-autoloads) prior to (require 'elpaca) does not help. The same thing seems to happen with the following bootstrap code.

(defvar elpaca-directory (expand-file-name "elpaca/" user-emacs-directory))
(defvar elpaca-builds-directory (expand-file-name "builds/" elpaca-directory))
(defvar elpaca-order `(elpaca :repo "https://github.com/progfolio/elpaca.git"
                  :ref "feat/elpaca-wait"
                  :build (:not elpaca--activate-package)))
(when-let ((repo  (expand-file-name "repos/elpaca/" elpaca-directory))
       (build (expand-file-name "elpaca/" elpaca-builds-directory))
       (order (cdr elpaca-order))
       ((add-to-list 'load-path (if (file-exists-p build) build repo)))
       ((not (file-exists-p repo))))
  (condition-case-unless-debug err
      (if-let ((buffer (pop-to-buffer-same-window "*elpaca-bootstrap*"))
           ((zerop (call-process "git" nil buffer t "clone"
                     (plist-get order :repo) repo)))
           (default-directory repo)
           ((zerop (call-process "git" nil buffer t "checkout"
                     (or (plist-get order :ref) "--")))))
      (progn
        (byte-recompile-directory repo 0 'force)
        (require 'elpaca)
        (and (fboundp 'elpaca-generate-autoloads)
         (elpaca-generate-autoloads "elpaca" repo))
        (kill-buffer buffer))
    (error "%s" (with-current-buffer buffer (buffer-string))))
    ((error)
     (warn "%s" err)
     (delete-directory repo 'recursive))))
(setq backtrace-line-length nil)
(require 'elpaca-autoloads)
(require 'elpaca)
(elpaca `(,@elpaca-order))
(dolist (recipe (oo-read-recipes)) (eval `(elpaca ,recipe) t))
(elpaca-wait)
;; (load (expand-file-name "config.el" user-emacs-directory))

The elpaca autoloads file does indeed seem to be empty.

;;; elpaca-autoloads.el --- automatically extracted autoloads  -*- lexical-binding: t -*-
;;
;;; Code:


(provide 'elpaca-autoloads)
;; Local Variables:
;; version-control: never
;; no-byte-compile: t
;; no-update-autoloads: t
;; coding: utf-8
;; End:
;;; elpaca-autoloads.el ends here

A warning buffer is created that warns buffer-or-marker-p, nil. And the backtrace buffer is empty.

[progfolio]

OK. What's the output of elpaca-version for that set up? You may have to manually require elpaca since that is an autoloaded command.

[Luis-Henriquez-Perez]

What's the output of elpaca-version for that set up?

elpaca-version output 008949e.

You may have to manually require elpaca since that is an autoloaded command.

By "manually" do you mean via eval-expression?

[progfolio]

elpaca-version output 008949e.

It should've printed much more information than that. The main thing I'm after is what version of Emacs is this occurring on? Is this Emacs 28?

By "manually" do you mean via eval-expression?

That or M-x load-library should work.

[Luis-Henriquez-Perez]

It should've printed much more information than that.

This is a picture of the output of elpaca-version followed by emacs-version.

Is this Emacs 28?

Yes it is.

That or M-x load-library should work.

So what steps exactly do you want me to take?

Right now I'm:

1. switching to a branch of my config that has the elpaca wait bootstrap code
2. tangling my config so that the init file contains the bootstrap code
3. calling elpaca-delete to delete elpaca
4. starting a new emacs instance with --debug-init

As I understand it after step 4, you'd like me to use M-x load-library to load elpaca. But what after that?

[progfolio]

You are on an outdated version of Elpaca. There was a fix which should hopefully take care of the empty autoloads file on Emacs 28 applied in 9ed120f.

[Luis-Henriquez-Perez]

You are on an outdated version of Elpaca.

Sorry, I had assumed that the bootstrap code install the latest version of elpaca-wait since I did not specify a commit hash. Actually, I'm not sure why it didn't. Right now I'm trying to get the right version of elpaca installed.

[progfolio]

The bootstrap code should take care of that if the elpaca repo is not there. Make sure both elpaca/repos/elpaca and elpca/builds/elpaca are gone prior to starting a fresh Emacs instance.

[Luis-Henriquez-Perez]

Ok. That seemed to be the issue. Somehow elpaca-delete did not entirely delete elpaca (maybe it didn't delete the build directory?) and I wasn't getting the proper version of elpaca when starting a new emacs instance. Running the below commands got rid of elpaca entirely, forcing it to be cloned from scratch.

~ $ rm -rf ~/.config/emacs/elpaca/repos/elpaca/
~ $ rm -rf ~/.config/emacs/elpaca/builds/elpaca/

Upon doing this, the latest commit from elpaca-wait was built and my emacs successfully started.

I note that it took a long time: 78.34 seconds. I'm not sure if this is normal for cloning and building one package--maybe it is. M-x elpaca-log was empty when I checked it.

[progfolio]

I'm glad it started. That means the autoloads are being generated properly.

I note that it took a long time: 78.34 seconds. I'm not sure if this is normal for cloning and building one package

That does seem very long. What package was it? Are there any :pre/:post builds for the package?

M-x elpaca-log was empty when I checked it.

You likely had the default search filter of the initial build "#unique !finished", which only shows package which are not finished. Make sure to hit s followed by RET in the log buffer after everything finishes to clear the filter. You should see the current filter in the header-line of the buffer. A filter of .* will show everything. I would need to see a log to determine where the time is being spent.

[Luis-Henriquez-Perez]

I'm glad it started.

Yea me too. Now I've switched to the elpaca-wait bootstrap as my main init file.

That does seem very long. What package was it?

It was elpaca itself.

You likely had the default search filter of the initial build "#unique !finished", which only shows package which are not finished.

Yes that was it.

Make sure to hit s followed by RET in the log buffer after everything finishes to clear the filter.

Ok. Did that this time.

Something different happened. Previously elpaca had installed successfully--it had just taken a long time. This time it failed. This is the log for elpaca.

elpaca                         cloning              fatal: Too many arguments.                                                       73.769849
elpaca                         cloning              usage: git clone [<options>] [--] <repo> [<dir>]                                 73.852442
elpaca                         cloning                  -v, --verbose         be more verbose                                        73.899639
elpaca                         cloning                  -q, --quiet           be more quiet                                          73.946969
elpaca                         cloning                  --progress            force progress reporting                               74.029120
elpaca                         cloning                  --reject-shallow      don't clone shallow repository                         74.076757
elpaca                         cloning                  -n, --no-checkout     don't create a checkout                                74.124649
elpaca                         cloning                  --bare                create a bare repository                               74.206963
elpaca                         cloning                  --mirror              create a mirror repository (implies bare)              74.255055
elpaca                         cloning                  -l, --local           to clone from a local repository                       74.337915
elpaca                         cloning                  --no-hardlinks        don't use local hardlinks, always copy                 74.385463
elpaca                         cloning                  -s, --shared          setup as shared repository                             74.433494
elpaca                         cloning                  --recurse-submodules[=<pathspec>]                                            74.516892
elpaca                         cloning                                        initialize submodules in the clone                     74.566150
elpaca                         cloning                  --recursive ...       alias of --recurse-submodules                          74.648518
elpaca                         cloning                  -j, --jobs <n>        number of submodules cloned in parallel                74.696086
elpaca                         cloning                  --template <template-directory>                                              74.743884
elpaca                         cloning                                        directory from which templates will be used            74.826459
elpaca                         cloning                  --reference <repo>    reference repository                                   74.874553
elpaca                         cloning                  --reference-if-able <repo>                                                   74.922920
elpaca                         cloning                                        reference repository                                   75.005330
elpaca                         cloning                  --dissociate          use --reference only while cloning                     75.053601
elpaca                         cloning                  -o, --origin <name>   use <name> instead of 'origin' to track upstream       75.136548
elpaca                         cloning                  -b, --branch <branch>                                                        75.184431
elpaca                         cloning                                        checkout <branch> instead of the remote's HEAD         75.232597
elpaca                         cloning                  -u, --upload-pack <path>                                                     75.315282
elpaca                         cloning                                        path to git-upload-pack on the remote                  75.363669
elpaca                         cloning                  --depth <depth>       create a shallow clone of that depth                   75.447206
elpaca                         cloning                  --shallow-since <time>                                                       75.495235
elpaca                         cloning                                        create a shallow clone since a specific time           75.543623
elpaca                         cloning                  --shallow-exclude <revision>                                                 75.626314
elpaca                         cloning                                        deepen history of shallow clone, excluding rev         75.674737
elpaca                         cloning                  --single-branch       clone only one branch, HEAD or --branch                75.757722
elpaca                         cloning                  --no-tags             don't clone any tags, and make later fetches not to follow them 75.805938
elpaca                         cloning                  --shallow-submodules  any cloned submodules will be shallow                  75.863108
elpaca                         cloning                  --separate-git-dir <gitdir>                                                  75.946302
elpaca                         cloning                                        separate git dir from working tree                     75.994816
elpaca                         cloning                  -c, --config <key=value>                                                     76.078593
elpaca                         cloning                                        set config inside the new repository                   76.126721
elpaca                         cloning                  --server-option <server-specific>                                            76.175303
elpaca                         cloning                                        option to transmit                                     76.258583
elpaca                         cloning                  -4, --ipv4            use IPv4 addresses only                                76.307176
elpaca                         cloning                  -6, --ipv6            use IPv6 addresses only                                76.390516
elpaca                         cloning                  --filter <args>       object filtering                                       76.438948
elpaca                         cloning                  --also-filter-submodules                                                     76.487336
elpaca                         cloning                                        apply partial clone filters to submodules              76.570442
elpaca                         cloning                  --remote-submodules   any cloned submodules will use their remote-tracking branch 76.619055
elpaca                         cloning                  --sparse              initialize sparse-checkout file to include only files at root 76.668143
elpaca                         cloning                  --bundle-uri <uri>    a URI for downloading bundles before fetching from origin remote 76.751056
elpaca                         failed                   --bundle-uri <uri>    a URI for downloading bundles before fetching from origin remote 76.889193
elpaca                         adding-remotes       Already on 'feat/elpaca-wait'                                                    76.938080
elpaca                         adding-remotes       Your branch is up to date with 'origin/feat/elpaca-wait'.                        76.987125
elpaca                         ref-checked-out      feat/elpaca-wait checked out                                                     77.070946
elpaca                         cloning-deps         Cloning Dependencies                                                             77.119425
elpaca                         cloning-deps         No external dependencies detected                                                77.203406
elpaca                         linking              Linking build files                                                              77.252029
elpaca                         linking              Build files linked                                                               77.340228
elpaca                         autoloads            Generating autoloads: /home/luis/.config/emacs/elpaca/builds/elpaca              77.389485
elpaca                         autoloads              INFO     Scraping files for elpaca-autoloads.el...                             77.745242
elpaca                         autoloads              INFO     Scraping files for elpaca-autoloads.el...done                         77.830541
elpaca                         autoloads            Autoloads Generated                                                              77.881383
elpaca                         byte-compilation     Byte compiling                                                                   77.967260
elpaca                         failed               elpaca--byte-compile: (wrong-type-argument number-or-marker-p nil)               78.165459

This is the initial error message I saw in the log. This same error happened to me to day when I tried deleting a package (consult) and re-installing it.

[progfolio]

Next time you get an error like that hit s and change the search filter to #verbosity. That will log the git commands that were issued as well as their output. It would also be useful to know what the output of git --version is on your system. It may be that a git command I'm using under the hood is not compatible with your version of git.

[Luis-Henriquez-Perez]

It would also be useful to know what the output of git --version is on your system

~/dotfiles $ git --version
git version 2.38.1

I typed in #verbosity after pressing s but the result looks the same. Is verbosity a search tag? I didn't see it in elpaca-ui-search-tags.

elpaca                         cloning              $git clone Elpaca Log (41 matches)  #unique !finished https://github.com/progfolio/elpaca.git /home/luis/.config/emacs/elpaca/repos/elpaca.github.progfolio/ 95.332654
elpaca                         cloning              fatal: Too many arguments.                                                       95.332709
elpaca                         cloning              usage: git clone [<options>] [--] <repo> [<dir>]                                 95.421294
elpaca                         cloning                  -v, --verbose         be more verbose                                        95.477433
elpaca                         cloning                  -q, --quiet           be more quiet                                          95.566016
elpaca                         cloning                  --progress            force progress reporting                               95.621910
elpaca                         cloning                  --reject-shallow      don't clone shallow repository                         95.711124
elpaca                         cloning                  -n, --no-checkout     don't create a checkout                                95.800044
elpaca                         cloning                  --bare                create a bare repository                               95.855675
elpaca                         cloning                  --mirror              create a mirror repository (implies bare)              95.946969
elpaca                         cloning                  -l, --local           to clone from a local repository                       96.008953
elpaca                         cloning                  --no-hardlinks        don't use local hardlinks, always copy                 96.105025
elpaca                         cloning                  -s, --shared          setup as shared repository                             96.199482
elpaca                         cloning                  --recurse-submodules[=<pathspec>]                                            96.258898
elpaca                         cloning                                        initialize submodules in the clone                     96.348403
elpaca                         cloning                  --recursive ...       alias of --recurse-submodules                          96.407318
elpaca                         cloning                  -j, --jobs <n>        number of submodules cloned in parallel                96.496499
elpaca                         cloning                  --template <template-directory>                                              96.586223
elpaca                         cloning                                        directory from which templates will be used            96.641989
elpaca                         cloning                  --reference <repo>    reference repository                                   96.731545
elpaca                         cloning                  --reference-if-able <repo>                                                   96.787768
elpaca                         cloning                                        reference repository                                   96.876675
elpaca                         cloning                  --dissociate          use --reference only while cloning                     96.966933
elpaca                         cloning                  -o, --origin <name>   use <name> instead of 'origin' to track upstream       97.022944
elpaca                         cloning                  -b, --branch <branch>                                                        97.112437
elpaca                         cloning                                        checkout <branch> instead of the remote's HEAD         97.168900
elpaca                         cloning                  -u, --upload-pack <path>                                                     97.259296
elpaca                         cloning                                        path to git-upload-pack on the remote                  97.349438
elpaca                         cloning                  --depth <depth>       create a shallow clone of that depth                   97.405284
elpaca                         cloning                  --shallow-since <time>                                                       97.494507
elpaca                         cloning                                        create a shallow clone since a specific time           97.551080
elpaca                         cloning                  --shallow-exclude <revision>                                                 97.640174
elpaca                         cloning                                        deepen history of shallow clone, excluding rev         97.730547
elpaca                         cloning                  --single-branch       clone only one branch, HEAD or --branch                97.786623
elpaca                         cloning                  --no-tags             don't clone any tags, and make later fetches not to follow them 97.876228
elpaca                         cloning                  --shallow-submodules  any cloned submodules will be shallow                  97.941695
elpaca                         cloning                  --separate-git-dir <gitdir>                                                  98.031743
elpaca                         cloning                                        separate git dir from working tree                     98.121758
elpaca                         cloning                  -c, --config <key=value>                                                     98.178377
elpaca                         cloning                                        set config inside the new repository                   98.269265
elpaca                         cloning                  --server-option <server-specific>                                            98.327187
elpaca                         cloning                                        option to transmit                                     98.418432
elpaca                         cloning                  -4, --ipv4            use IPv4 addresses only                                98.509313
elpaca                         cloning                  -6, --ipv6            use IPv6 addresses only                                98.566220
elpaca                         cloning                  --filter <args>       object filtering                                       98.658933
elpaca                         cloning                  --also-filter-submodules                                                     98.720159
elpaca                         cloning                                        apply partial clone filters to submodules              98.816915
elpaca                         cloning                  --remote-submodules   any cloned submodules will use their remote-tracking branch 98.913124
elpaca                         cloning                  --sparse              initialize sparse-checkout file to include only files at root 98.970319
elpaca                         cloning                  --bundle-uri <uri>    a URI for downloading bundles before fetching from origin remote 99.060651
elpaca                         failed                   --bundle-uri <uri>    a URI for downloading bundles before fetching from origin remote 99.211251