search

progfolio / elpaca

An elisp package manager
GNU General Public License v3.0
639 stars 31 forks source link

installing packages from a list of recipes #28

Closed Luis-Henriquez-Perez closed 1 year ago

Luis-Henriquez-Perez commented 1 year ago

background

I am in the process of migrating from straight.el. The way I used straight is unconventional. I used an elisp script to register a list of recipes with straight-register-package. I am trying to replicate this process with elpaca.

I have an org file with elpaca acceptable recipes for the packages I adapted from my straight recipes. Each heading in the file has a source block with a package recipe. I display the first one as an example below.

* evil-goggles
:PROPERTIES:
:LABELS:   [[id:e80a07f6-a377-4f86-b892-d221897e8980][recipe]]
:ID:       20cd3665-1f17-4a1a-b30f-ef428e192862
:END:
#+begin_src emacs-lisp
'(evil-goggles :host github :repo "edkolev/evil-goggles" :branch nil :depth 1 :ref "08a22058fd6a167f9f1b684c649008caef571459")
#+end_src

what I envision achieving

So in essense what I want is for a package manager to simply install a list of packages into a directory that I specify. And to provide me with directories I can append to the load-path, custom-theme-directory and Info-directory-files. I don't want my init file to install any packages, I just want it to assume these directories exist and append to these variables accordingly.

my thought process

I thought about just going down the list and calling elpaca on each recipe one by one, but some packages are dependencies of others. And installing a dependency might pull in other packages before I've had a chance to specify their recipe. For example, evil-goggles is (by chance) the first item in my list of recipes. But calling elpaca on it might install evil.

I recall straight had straight-register-package for this reason which would register package recipes before installing them.

what I tried

I tried using elpaca-recipe-functions for this. (I also tried adding to elpaca-order-functions, not sure which one to use). The idea being that I wanted merge plist values to the built-in recipes so that my preferences for :ref and :repo and :depth were respected.

;; create
;; An alist of elpaca recipes.
(defun oo-elpaca-custom-recipes ()
  "Return recipes in recipes.org as an alist."
  (with-temp-buffer
    (insert-file-contents "~/dotfiles/recipes.org")
    (goto-char (point-min))
    (let (orders)
      (while (re-search-forward (rx "#+begin_src emacs-lisp\n" (group-n 1 (1+ not-newline)) "\n#+end_src") nil t)
    (push (cadr (car (read-from-string (match-string 1)))) orders))
      orders)))

(defvar oo-elpaca-custom-recipes (oo-elpaca-custom-recipes)
  "Custom recipes derived from recipe files.")

(defun oo-elpaca-custom-recipes-fn (recipe)
  (alist-get (car recipe) oo-elpaca-custom-orders))

(add-hook 'elpaca-recipe-functions #'oo-elpaca-custom-recipes-fn)

;; I think this should work instead of the ugliness I did below.
;; (mapc #'elpaca-try (mapcar #'car oo-elpaca-custom-recipes))

(let* ((packages (mapcar #'car oo-elpaca-custom-recipes))
       (elpaca-forms (mapcar (lambda (package) `(elpaca ,package (message "Installed %s..." ',package))) packages)))
  (funcall `(lambda () ,@elpaca-forms)))

This didn't work.

second attempt

I try relying on elpaca merging the recipe I provide with its internal one to do what I mean.

(defun oo-elpaca-custom-orders ()
  "Try to see if I have a recipe registered."
  (with-temp-buffer
    (insert-file-contents "~/dotfiles/recipes.org")
    (goto-char (point-min))
    (let (orders)
      (while (re-search-forward (rx "#+begin_src emacs-lisp\n" (group-n 1 (1+ not-newline)) "\n#+end_src") nil t)
    (push (cadr (car (read-from-string (match-string 1)))) orders))
      orders)))

(defvar oo-elpaca-custom-orders (oo-elpaca-custom-orders)
  "Custom recipes derived from recipe files.")

(let* ((packages oo-elpaca-custom-orders)
       (elpaca-forms (mapcar (lambda (package) `(elpaca ,package (message "Installed %s..." ',package))) packages)))
  (funcall `(lambda () ,@elpaca-forms)))

This is successfully installing packages, but I'm not sure if my concern about registering packages/sorting by dependencies is being met. Maybe it is because elpaca works asyncronously so I'm not sure if it is actually installing packages yet until elpaca-process-queues is called.

result of what I tried

Many packages were installed, as claimed by elpaca-status. However, the queue processing seemed to stop on sevaral occations, I had to manually call elpaca-process-queues multiple times. I'm not completely sure what's happening.

progfolio commented 1 year ago

I thought about just going down the list and calling elpaca on each recipe one by one, but some packages are dependencies of others. And installing a dependency might pull in other packages before I've had a chance to specify their recipe.

The elpaca macro queues orders. So long as all of the orders are queued prior to the queue being processed (via elpaca-process-queues), explicitly requested orders will take precedence over implicit dependencies. Workarounds which involve hoisting recipe declarations in straight.el (e.g. (straight-use-package 'org) prior to anything which depends on Org) should not be necessary in Elpaca.

Regarding your desired usage, you should be able to queue all your orders via a loop. Something like this should work, assuming the my-orders function returns a valid list of orders.

;; Queue all orders returned by `my-orders' function
(dolist (order (my-orders)) (eval `(elpaca ,order) t))
;; Process the queue
(elpaca-process-queues)

I tried using elpaca-recipe-functions for this. (I also tried adding to elpaca-order-functions, not sure which one to use).

I try relying on elpaca merging the recipe I provide with its internal one to do what I mean.

It sounds to me like you want recipe inheritance here. Neither hook needs to be modified for that to work. It is turned on by default via elpaca-order-defaults.

I just want it to assume these directories exist and append to these variables accordingly.

Managing the load-path, Info dirs, etc. is all functionality provided by Elpaca. You could eschew that, but it seems like it would lead to a fragile configuration. Could you elaborate on the benefit of the setup you have in mind?

Luis-Henriquez-Perez commented 1 year ago

Regarding your desired usage, you should be able to queue all your orders via a loop. Something like this should work, assuming the my-orders function returns a valid list of orders.

Cool. I tried doing this. I noticed that the order processing would stop. As in, it all the packages wouldn't get installed. I want to be able to bootstrap my configuration smoothly, so it's important to me that I can get all the packages installed successfully in one run.

I tried adding message information so I could see what was going on.

The last message I get is "BEGINNING TO PROCESS QUEUES". I don't get any "DONE WITH ORDER" messages. (Note that before the following code I have the exact bootstrap snippet from the readme which I didn't include here for brevity).

(defun oo-elpaca-custom-orders ()
  "Try to see if I have a recipe registered."
  (with-temp-buffer
    (insert-file-contents "~/dotfiles/recipes.org")
    (goto-char (point-min))
    (let (orders)
      (while (re-search-forward (rx "#+begin_src emacs-lisp\n" (group-n 1 (1+ not-newline)) "\n#+end_src") nil t)
    (push (cadr (car (read-from-string (match-string 1)))) orders))
      orders)))

(defvar oo-elpaca-custom-orders (oo-elpaca-custom-orders)
  "Custom recipes derived from recipe files.")

(message "BEGINNING TO REGISTER ORDERS..")

(dolist (order (oo-elpaca-custom-orders))
  (eval `(elpaca ,order (message "DONE WITH ORDER: %s" ',(car order))) t))

(add-hook 'after-init-hook (lambda () (message "BEGINNING TO PROCESS QUEUES...") (elpaca-process-queues)))

(add-hook 'elpaca-after-init-hook (lambda (&rest _) "DONE INSTALLING PACAKGES..."))

If an order is invalid, will it prevent other orders from being processed?

Could you elaborate on the benefit of the setup you have in mind?

Primarily performance. You could also argue more abstractly a separation of concerns as well, package installation vs configuration.

My thinking is that if a package manager could provide a function installs packages based on given recipes into a directory and generate files/directories that correspond to the load-path, info-dirs, etc. such that the user doesn't have to require the package manager on startup. He just points the corresponding variables to the corresponding places. If package management at startup could be reduced to to appending paths to some variables, then emacs would startup super fast. When you need to install, remove packages you could just require the package manager.

This is, of course, in contrast to the way most emacs configurations work, where packages are installed and configured together.

I don't know how feasible this is. I don't see why it's not possible.

In case you are curious I will mention that in the setup I am migrating from, I did this with straight. I didn't even autoload any functions. I created my own autoloading mechanism where I guessed the parent feature of a function based on its name. Maybe it sounds crazy but it turned out pretty well! :) Though you do have to account for some "abnormal" packages whose function prefix doesn't match their name (such as expand-region whose functions are prefixed by "er/").

(defun! oo-possible-features (fn)
  "Return a list of possible features available for FN."
  (expr! fname (symbol-name fn))
  (dolist (path load-path)
    (expr! base (file-name-sans-extension (file-name-nondirectory (directory-file-name path))))
    (when (s-prefix-p base fname)
      (collecting! possible (intern base))))
  (dolist! ((prefix . feature) oo-abnormal-feature-alist)
    (when (s-prefix-p (oo-args-to-string prefix) fname)
      (collecting! features feature)))
  (or (seq-sort-by (-compose #'length #'symbol-name) #'> possible)
      features))

(defun oo-autoload-fn (fn &optional feature)
  "If FN is bound return FN, otherwise return an interactive lambda."
  (unless (and (symbolp fn) (fboundp fn))
    (alet `(lambda (&rest _)
         (interactive)
         (if-let (feature (or ',feature (car (oo-possible-features #',fn))))
         (progn (fmakunbound #',fn)
            (oo--log-info "Autoloading %s from %s" #',fn feature)
            (require feature)
            (cond ((fboundp #',fn)
                   (alet (symbol-function #',fn)
                 (if (keymapp it)
                     (set-transient-map it)
                   (call-interactively #',fn))))
                  (t
                   (error "Not able to load %s from %s." #',fn feature))))
           (error "Not able to find feature for %s." #',fn)))
      (fset fn it)))
  fn)

You could eschew that, but it seems like it would lead to a fragile configuration.

You are probably correct about this. Considering this point, I'd be happy if I could just require elpaca to setup the load-path and possibly activate autoloads. Right now as I'm migrating I'm primarly just focused on getting the bootstrapping working consistently.

progfolio commented 1 year ago

Cool. I tried doing this. I noticed that the order processing would stop. As in, it all the packages wouldn't get installed. I want to be able to bootstrap my configuration smoothly, so it's important to me that I can get all the packages installed successfully in one run.

I tried adding message information so I could see what was going on.

Elpaca should log every package event. The output of M-x elpaca-log should give you a clue as to what is happening.

If an order is invalid, will it prevent other orders from being processed?

In most cases the order will just immediately fail and show as failed in the log along with the error that caused it to fail. The queue should still finalize despite an order failing. An order that is blocked will prevent the queue from finalizing. Please share the output of elpaca-log so I can see what's happening.

Primarily performance.

I'm aiming to keep the overhead of Elpaca relatively low.

I'd be happy if I could just require elpaca to setup the load-path and possibly activate autoloads.

On sbusequent inits after the initial install, that's essentially what Elpaca does. It:

I've profiled start up time on my machine while developing Elpaca and (in my case) the overhead added by queues and logging is negligible, especially considering the diagnostic benefit of the logging. The bulk of the time is spent loading/reading package configurations (independent of whether Elpaca, straight, etc are used). If you notice anything that you feel is too slow, please let me know and I can look into optimizing it.

You could also argue more abstractly a separation of concerns as well, package installation vs configuration.

Regarding separating concerns, if your init file is written in a way that it depends on directories, features, etc being installed it makes sense that the installation and configuration would "live together" in one declaration. use-package encourages this with package.el via the :ensure keyword. Otherwise you'd have to write your init in a way that guards against the possibility of something not being installed (and then deal with that elsewhere when you do want to install). IMO it's cleaner to consider these concerns together.

Luis-Henriquez-Perez commented 1 year ago

This is the display of elpaca-log. It's at this point I stop seeing activity. The last message (of the ones I print myself) I see in the messages buffer is "BEGINNING TO PROCESS QUEUES...", which is the message I set to print just before calling elpaca-process-queues. I see the void variable error about auto-save-list-file, could that be the problem?

Package Status Info Time ▼ 
ivy                            blocked              Waiting on monorepo "/home/luis/.config/emacs/elpaca/repos/swiper"               12.130661
hydra                          blocked              Waiting on monorepo "/home/luis/.config/emacs/elpaca/repos/hydra"                12.131489
git-commit                     blocked              Waiting on monorepo "/home/luis/.config/emacs/elpaca/repos/magit"                12.135017
dash                           blocked              Waiting on monorepo "/home/luis/.config/emacs/elpaca/repos/dash"                 12.235103
counsel                        blocked              Waiting on monorepo "/home/luis/.config/emacs/elpaca/repos/swiper"               12.235427
emms                           cloning              Checking out ref: 5c3226bec64bc5ad6a496b1619144087ba400481                       40.398848
dash-functional                cloning              Checking out ref: 7a9c9378772b687a452966ce4745c54afb19a2fc                       40.412230
org-super-links                cloning              Checking out ref: 01fb73264a399143a79bb2c68d9b4dd868ddb052                       40.422566
all-the-icons-completion       cloning              Checking out ref: 286e2c064a1298be0d8d4100dc91d7a7a554d04a                       40.439987
embark                         cloning              Checking out ref: 5d0459d27aa7cf738b5af36cf862723a62bef955                       40.452261
zoom-window                    blocked              Checking out ref: 474ca4723517d95356145950b134652d5dc0c7f7                       40.476143
zoom-frm                       blocked              Checking out ref: 59e2fced1819e98acc92da93d8a22789f084d697                       40.482755
xr                             blocked              Checking out ref: 277c5490d554ee3fe3e99e53d28a78a5fc3329c8                       40.495698
workgroups2                    blocked              Checking out ref: c9403c68a7e6491134110d7cacc130c34eae85a0                       40.516010
with-editor                    blocked              Checking out ref: 139ef3933ea7aa3fe67b87450a6a1ac0895e5c81                       40.518666
which-key                      blocked              Checking out ref: 428aedfce0157920814fbb2ae5d00b4aea89df88                       40.531998
undo-tree                      blocked              Checking out ref: e326c6135e62f5fe8536528d3acd5e798f847407                       40.554859
ts                             blocked              Checking out ref: b7ca357a0ed57694e0b25ec1b1ca12e24a4ce541                       40.568102
transient                      blocked              Checking out ref: 90e640fe8fa3f309c7cf347501e86ca5cd0bd85e                       40.575999
toc-org                        blocked              Checking out ref: aef220c266f53d36055f74f4a243c6483c563d2a                       40.582778
swiper                         blocked              Checking out ref: 8f2abd397dba7205806cfa1615624adc8cd5145f                       40.636981
super-save                     blocked              Checking out ref: 886b5518c8a8b4e1f5e59c332d5d80d95b61201d                       40.649833
smartparens                    blocked              Checking out ref: 63695c64233d215a92bf08e762f643cdb595bdd9                       40.676086
separedit                      blocked              Checking out ref: dc0b3448f3d9738f5233c34c5c8fc172eda26323                       40.684518
s                              blocked              Checking out ref: 43ba8b563bee3426cead0e6d4ddc09398e1a349d                       40.694603
rainbow-delimiters             blocked              Checking out ref: f43d48a24602be3ec899345a3326ed0247b960c6                       40.720010
prescient                      blocked              Checking out ref: 42adc802d3ba6c747bed7ea1f6e3ffbbdfc7192d                       40.734320
ppp                            blocked              Checking out ref: 86dad69c3a7dae770f6b99285647dff2aad81930                       40.744092
popup                          blocked              Checking out ref: bd5a0df7e5bc68af46eef37afe9e80764a1d4fd8                       40.755970
password-store-otp             blocked              Checking out ref: 04998c8578a060ab4a4e8f46f2ee0aafad4ab4d5                       40.784108
password-store                 blocked              Checking out ref: f152064da9832d6d3d2b4e75f43f63bf2d50716f                       40.791809
org-superstar                  blocked              Checking out ref: 7f83636db215bf5a10edbfdf11d12a132864a914                       40.806193
org-super-agenda               blocked              Checking out ref: f5e80e4d0da6b2eeda9ba21e021838fa6a495376                       40.816633
org-journal                    blocked              Checking out ref: 08d5fce95023c015372678d353388ad0dae8952b                       40.844588
orderless                      blocked              Checking out ref: cbc0109eac542ef4fe0be027af1c62c4bbf846ee                       40.848058
mmt                            blocked              Checking out ref: d7729563e656a3e8adef6bce60348861ba183c09                       40.853888
mini-modeline                  blocked              Checking out ref: 7dcd0ab81bb7c298377708061176f5c5a50f77db                       40.861069
map                            blocked              Checking out ref: dc4f657bcce6ec644ebf96fe52d8035aa33882c0                       40.881835
lispyville                     blocked              Checking out ref: 0f13f26cd6aa71f9fd852186ad4a00c4294661cd                       41.019963
lispy                          blocked              Checking out ref: 1ad128be0afc04b58967c1158439d99931becef4                       41.028102
keyfreq                        blocked              Checking out ref: e5fe9d585ce882f1ba9afa5d894eaa82c79be4f4                       41.037469
ht                             blocked              Checking out ref: 2850301d19176b8d3bb6cc8d95af6ab7e529bd56                       41.096075
helpful                        blocked              Checking out ref: 584ecc887bb92133119f93a6716cdf7af0b51dca                       41.134920
grugru                         blocked              Checking out ref: 92e588e9749614ef6cb68b76b1d3aaadf7731406                       41.159963
goto-chg                       blocked              Checking out ref: 2af612153bc9f5bed135d25abe62f46ddaa9027f                       41.165546
gif-screencast                 blocked              Checking out ref: 1145e676b160e7b1e5756f5b0f30dd31de252e1f                       41.181939
frame-cmds                     blocked              Checking out ref: b803354c8cf7c9aafcea1ff4e67288bea0719599                       41.204600
f                              blocked              Checking out ref: 1814209e2ff43cf2e6d38c4cd476218915f550fb                       41.239953
exwm-edit                      blocked              Checking out ref: 2fd9426922c8394ec8d21c50dcc20b7d03af21e4                       41.285808
elfeed-org                     blocked              Checking out ref: 77b6bbf222487809813de260447d31c4c59902c9                       41.346661
el-mock                        blocked              Checking out ref: 5df1d3a956544f1d3ad0bcd81daf47fff33ab8cc                       41.356248
doct                           blocked              Checking out ref: 15974ad8d4d7baa071b5ea33877e9dc117c4153e                       41.364151
avy                            blocked              Checking out ref: e92cb37457b43336b765630dbfbea8ba4be601fa                       41.405163
auth-source-pass               blocked              Checking out ref: aa7f17116ec3f760eb414d655ba20016b11a4a0e                       41.419375
ace-window                     blocked              Checking out ref: c7cb315c14e36fded5ac4096e158497ae974bec9                       41.452083
with-emacs                     blocked              Checking out ref: 9f99bec56f87e53deb9f33b364eda77677a17eb9                       41.465183
org-ql                         blocked              Checking out ref: d7ada532c7d06e91d6e07800ca22d5fbdb970e3e                       46.400873
magit                          cloning              Checking out ref: 86eec7ba39eb46fa1e4c2f37800d22c6dfd155c7                       46.407551
lv                             cloning              Checking out ref: 2d553787aca1aceb3e6927e426200e9bb9f056f1                       46.415809
exwm-firefox-evil              cloning              Checking out ref: 14643ee53a506ddcb5d2e06cb9f1be7310cd00b1                       46.422814
expand-region                  cloning              Checking out ref: ea6b4cbb9985ddae532bd2faf9bb00570c9f2781                       46.426760
elisp-refs                     cloning              Checking out ref: b3634a4567c655a1cda51b217629849cba0ac6a7                       46.449962
elisp-demos                    cloning              Checking out ref: ed9578dfdbbdd6874d497fc9873ebfe09f869570                       46.468597
elfeed                         cloning              Checking out ref: de4b64b3f5d9fd41d9dc72023632ae535dc912e2                       46.499982
el-get                         blocked              Checking out ref: 84dd1837f9ac80a329ab0c2de6859777f445f8ff                       46.514142
dashboard                      cloning              Checking out ref: 36c8da41bca977707c42bd9b13cdf39380b957d7                       46.520881
corfu                          cloning              Checking out ref: a59c41d90ede24cf5cdf4104790af3c318174b08                       46.528963
consult                        blocked              Checking out ref: 822928a8609730e8c22e068b04d7908312706cfd                       46.542523
bind-key                       cloning              Checking out ref: 365c73d2618dd0040a32c2601c5456ab5495b812                       46.555309
async                          cloning              Checking out ref: 9a8cd0c3d5c120bfa03187c54dba6e33f6e3ca19                       46.564272
aggressive-indent              cloning              Checking out ref: b0ec0047aaae071ad1647159613166a253410a63                       46.585192
redacted                       cloning              Checking out ref: 156311eb128bd6cc7d6912552c70f6cf22a14cff                       46.745927
tempel                         cloning              Checking out ref: b4bb7030e9fa4a9451b79dfb2e815e0dd796527d                       46.756237
ednc                           cloning              Checking out ref: d1a3c37235dd87e0bce6ffc75f5568218d6d83b4                       46.768382
svg-lib                        cloning              Checking out ref: 0486c9453449771bc3f5872f70bc5cb23580d0f4                       46.788245
svg-tag-mode                   cloning              Checking out ref: fee61c6a0b0570bd24fd335efef17c7385297aa0                       46.792299
ef-themes                      cloning              Checking out ref: 4555374a965175094b6d1ed770f53d713550afe7                       46.837826
spacemacs-theme                cloning              Checking out ref: 1f5b03254de6bfa9645711f2b79781f5cca8d203                       46.848266
evil-collection                cloning              Checking out ref: 2d3d652cb51eeddc6c63ad9cbf251ecbd2f561d6                       46.860345
aggressive-fill-paragraph      cloning-deps         Cloning Dependencies                                                             46.891939
dbc                            cloning-deps         Cloning Dependencies                                                             47.029813
evil-easymotion                cloning-deps         Cloning Dependencies                                                             47.232177
evil-goggles                   cloning-deps         Cloning Dependencies                                                             47.239971
evil-surround                  cloning-deps         Cloning Dependencies                                                             47.241024
exwm-firefox-core              cloning-deps         Cloning Dependencies                                                             47.255987
org-ml                         cloning-deps         Cloning Dependencies                                                             48.232309
origami                        cloning-deps         Cloning Dependencies                                                             48.264132
eshell-up                      cloning              Checking out ref: 9c100bae5c3020e8d9307e4332d3b64e7dc28519                       51.144675
evil                           blocked              Checking out ref: cc9d6886b418389752a0591b9fcb270e83234cf9                       57.412016
elpaca                         byte-compilation     Symbol’s value as variable is void: auto-save-list-file-prefix                   57.432059
dirvish                        cloning-deps         Cloning Dependencies                                                             60.231932
elfeed-score                   cloning-deps         Cloning Dependencies                                                             60.280624
ellocate                       cloning-deps         Cloning Dependencies                                                             60.331944
pdf-tools                      cloning              Checking out ref: c510442ab89c8a9e9881230eeb364f4663f59e76                       60.348531
pass                           cloning              Checking out ref: a095d24cf06a7b0fbc3add480c101304a91cf788                       60.371934
modus-themes                   blocked              Checking out ref: 38236a925ef34f8e8c51babee587b594e77dffbe                       60.403953
exwm                           cloning              Checking out ref: 45ac28cc9cffe910c3b70979bc321a1a60e002ea                       60.464000
evil-magit                     cloning              Checking out ref: f4a8c8d3a5a699baea9356be7c1c5fd8867f610c                       60.508076
engine-mode                    cloning              Checking out ref: e0910f141f2d37c28936c51c3c8bb8a9ca0c01d1                       60.543947
swiper-helm                    blocked              Blocked by dependencies: (swiper helm)                                           62.488050
all-the-icons                  blocked              Checking out ref: be99987eda1ba3fdc490984b207849689599b858                       65.611991
git-gutter+                    cloning-deps         Cloning Dependencies                                                             65.616245
emojify                        blocked              Checking out ref: cfa00865388809363df3f884b4dd554a5d44f835                       68.930353
iedit                          byte-compilation     Symbol’s value as variable is void: auto-save-list-file-prefix                   69.037074
helm                           cloning-deps         Cloning Dependencies                                                             72.914760
helm-core                      cloning-deps         Cloning Dependencies                                                             72.924220
evil-cleverparens              blocked              Blocked by dependencies: (evil smartparens dash)                                 72.928637
loopy-dash                     blocked              Blocked by dependencies: (dash)                                                  75.095008
exwm-float                     blocked              Blocked by dependencies: (exwm)                                                  79.366711
org                            blocked              Checking out ref: d012350dadf15835e361626ba0a6ec08f8c7a715                       80.683552
progfolio commented 1 year ago

Thank you for sharing the log.

I see the void variable error about auto-save-list-file, could that be the problem?

That does look suspicious. Especially considering it appears in two different spots. If you launch emacs with the --debug-init option you should get a backtrace when those errors are signaled. It would be helpful if we could see that as well.

Luis-Henriquez-Perez commented 1 year ago

If you launch emacs with the --debug-init option you should get a backtrace when those errors are signaled.

It was already with --debug-init enabled. Maybe it didn't raise an error because it was during byte-compilation and the byte compilation is done asynchronously?

What I don't get is that elpaca processing seems to just stop without apparent reason. At first I kept waiting because I didn't realize it stopped. I had a time where I installed almost all the packages, but that was manually re-evaluating (elpaca-process-queue) via eval-expression.

progfolio commented 1 year ago

It was already with --debug-init enabled. Maybe it didn't raise an error because it was during byte-compilation and the byte compilation is done asynchronously?

Interesting. The debugger should still kick in if an error is signaled in the process sentinel.

What's the output of M-x emacs-version? Operating system? Are you able to share the init file you are using to get this output, too?

Luis-Henriquez-Perez commented 1 year ago

What's the output of M-x emacs-version?

GNU Emacs 27.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, cairo version 1.16.0)

Operating System?

Guix System 371aa5777a3805a3886f3feea5f1960fe3fe4219 x86_64

My os is an outdated version of GUIX. I am meaning to switch to arch or upgrade to the newest version of GUIX. But I want to have a solid, bootstrapable emacs config before I do that.

Are you able to share the init file you are using to get this output, too?

This is the exact content of my init file. I deleted everything except this code and still get the same issue.

(message "BOOTSTRAP CODE...")

(declare-function elpaca-generate-autoloads "elpaca")
(defvar elpaca-directory (expand-file-name "elpaca/" user-emacs-directory))
(defvar elpaca-builds-directory (expand-file-name "builds/" elpaca-directory))
(when-let ((elpaca-repo (expand-file-name "repos/elpaca/" elpaca-directory))
       (elpaca-build (expand-file-name "elpaca/" elpaca-builds-directory))
       (elpaca-target (if (file-exists-p elpaca-build) elpaca-build elpaca-repo))
       (elpaca-url  "https://www.github.com/progfolio/elpaca.git")
       ((add-to-list 'load-path elpaca-target))
       ((not (file-exists-p elpaca-repo)))
       (buffer (get-buffer-create "*elpaca-bootstrap*")))
  (condition-case-unless-debug err
      (progn
    (unless (zerop (call-process "git" nil buffer t "clone" elpaca-url elpaca-repo))
      (error "%s" (list (with-current-buffer buffer (buffer-string)))))
    (byte-recompile-directory elpaca-repo 0 'force)
    (require 'elpaca)
    (elpaca-generate-autoloads "elpaca" elpaca-repo)
    (kill-buffer buffer))
    ((error)
     (delete-directory elpaca-directory 'recursive)
     (with-current-buffer buffer
       (goto-char (point-max))
       (insert (format "\n%S" err))
       (display-buffer buffer)))))
(require 'elpaca-autoloads)
(elpaca (elpaca :host github :repo "progfolio/elpaca"))

(defun oo-elpaca-custom-orders ()
  "Try to see if I have a recipe registered."
  (with-temp-buffer
    (insert-file-contents "~/dotfiles/recipes.org")
    (goto-char (point-min))
    (let (orders)
      (while (re-search-forward (rx "#+begin_src emacs-lisp\n" (group-n 1 (1+ not-newline)) "\n#+end_src") nil t)
    (push (cadr (car (read-from-string (match-string 1)))) orders))
      orders)))

(defvar oo-elpaca-custom-orders (oo-elpaca-custom-orders)
  "Custom recipes derived from recipe files.")

(message "BEGINNING TO REGISTER ORDERS..")

(dolist (order (oo-elpaca-custom-orders))
  (message "ordering %s" (car order))
  (eval `(elpaca ,order (message "DONE WITH ORDER: %s" ',(car order))) t))

(add-hook 'after-init-hook (lambda () (message "BEGINNING TO PROCESS QUEUES...") (elpaca-process-queues)))

(add-hook 'elpaca-after-init-hook (lambda (&rest _) "DONE INSTALLING PACAKGES..."))

The first two recipes returned by (oo-elpaca-custom-orders):

(zoutline :fetcher github :repo "abo-abo/zoutline" :depth 1 :ref "63756846f8540b6faf89d885438186e4fe1c7d8a")
(zoom-window :fetcher github :repo "emacsorphanage/zoom-window" :depth 1 :ref "474ca4723517d95356145950b134652d5dc0c7f7")

If I had to guess, its a recipe problem. But all of my recipes are prety much the same format as this.

Luis-Henriquez-Perez commented 1 year ago

recipes.txt

This is the recipes file, which I sent as an org file because github doesn't support org file attatchments.

init.txt

And my init file as a txt file for the same reason.

Luis-Henriquez-Perez commented 1 year ago

This might help too. It is the output of the Messages buffer when running my init file--and bootstrapping elpaca from scratch. I make sure to always delete the elpaca directory and its contents before running again. Notable is that no order seems to have been finished, because I don't see any of the "DONE WITH ORDER..." messages that I specified in the init file. Also, elpaca-process-queues seems to have ben called as it should have been because the message (message "BEGINNING TO PROCESS QUEUES...") which was in the same anonymous function was evaluated.

Checking /home/luis/.config/emacs/elpaca/repos/elpaca/...
Compiling /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca-log.el...done
Wrote /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca-log.elc
Checking /home/luis/.config/emacs/elpaca/repos/elpaca/...
Compiling /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca-manager.el...done
Wrote /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca-manager.elc
Checking /home/luis/.config/emacs/elpaca/repos/elpaca/...
Compiling /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca-menu-gnu-elpa-mirror.el...done
Wrote /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca-menu-gnu-elpa-mirror.elc
Checking /home/luis/.config/emacs/elpaca/repos/elpaca/...
Compiling /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca-menu-melpa.el...done
Wrote /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca-menu-melpa.elc
Checking /home/luis/.config/emacs/elpaca/repos/elpaca/...
Compiling /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca-menu-non-gnu-elpa.el...done
Wrote /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca-menu-non-gnu-elpa.elc
Checking /home/luis/.config/emacs/elpaca/repos/elpaca/...
Compiling /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca-menu-org.el...done
Wrote /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca-menu-org.elc
Checking /home/luis/.config/emacs/elpaca/repos/elpaca/...
Compiling /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca-process.el...done
Wrote /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca-process.elc
Checking /home/luis/.config/emacs/elpaca/repos/elpaca/...
Compiling /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca-ui.el...done
Wrote /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca-ui.elc
Checking /home/luis/.config/emacs/elpaca/repos/elpaca/...
Compiling /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca.el...done
Wrote /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca.elc
Checking /home/luis/.config/emacs/elpaca/repos/elpaca/...
Checking /home/luis/.config/emacs/elpaca/repos/elpaca/images...
Checking /home/luis/.config/emacs/elpaca/repos/elpaca/test... [3 times]
Done (Total of 9 files compiled, 2 skipped in 2 directories)
INFO     Scraping files for elpaca-autoloads.el... 
Generating autoloads for elpaca-log.el...done
Generating autoloads for elpaca-manager.el...done
Generating autoloads for elpaca-menu-gnu-elpa-mirror.el...done
Generating autoloads for elpaca-menu-melpa.el...done
Generating autoloads for elpaca-menu-non-gnu-elpa.el...done
Generating autoloads for elpaca-menu-org.el...done
Generating autoloads for elpaca-process.el...done
Generating autoloads for elpaca-ui.el...done
Generating autoloads for elpaca.el...done
INFO     Scraping files for elpaca-autoloads.el...done
Wrote /home/luis/.config/emacs/elpaca/repos/elpaca/elpaca-autoloads.el [2 times]
Downloading MELPA recipes...
Added to /home/luis/.config/emacs/elpaca/cache/melpa/.git/info/sparse-checkout
MELPA recipes downloaded.
MELPA menu index cached
Downloading GNU ELPA recipes...
GNU ELPA recipes downloaded.
Contacting host: elpa.gnu.org:443
Contacting host: elpa.nongnu.org:443
Downloading NonGNU ELPA recipes
NonGNU ELPA recipes downloaded
Downloading NonGNU ELPA recipes
NonGNU ELPA recipes downloaded
BEGINNING TO REGISTER ORDERS..
ordering zoutline
ordering zoom-window
ordering zoom-frm
ordering xr
ordering xelb
ordering workgroups2
ordering with-editor
ordering which-key
ordering vertico-quick
ordering vertico-buffer
ordering vertico
ordering undo-tree
ordering ts
ordering transpose-frame
ordering transient
ordering toc-org
ordering tablist
ordering system-packages
ordering swiper-helm
ordering swiper
ordering super-save
ordering spell-number
ordering smartparens
ordering shut-up
ordering separedit
ordering s
ordering restart-emacs
ordering rainbow-delimiters
ordering prescient
ordering prefixed-core
ordering ppp
ordering popwin
ordering popup
ordering plural
ordering pinentry
ordering peg
ordering pdf-tools
ordering password-store-otp
ordering password-store
ordering pass
ordering ov
ordering origami
ordering org-transclusion
ordering org-superstar
ordering org-super-agenda
ordering org-ql
ordering org-ml
ordering org-link-minor-mode
ordering org-journal
ordering org
ordering orderless
ordering modus-themes
ordering mmt
ordering mini-modeline
ordering memoize
ordering marginalia
ordering map
ordering magit
ordering macrostep
ordering lv
ordering loopy-iter
ordering loopy-dash
ordering loopy
ordering log4e
ordering lispyville
ordering lispy
ordering keyfreq
ordering key-chord
ordering ivy
ordering idle-require
ordering hydra
ordering ht
ordering highlight-quoted
ordering hide-mode-line
ordering helpful
ordering grugru
ordering goto-chg
ordering git-gutter+
ordering git-commit
ordering git-auto-commit-mode
ordering gif-screencast
ordering gcmh
ordering frame-fns
ordering frame-cmds
ordering fortune-cookie
ordering figlet
ordering f
ordering exwm-float
ordering exwm-firefox-evil
ordering exwm-firefox-core
ordering exwm-edit
ordering exwm
ordering expand-region
ordering evil-surround
ordering evil-magit
ordering evil-goggles
ordering evil-easymotion
ordering evil
ordering eshell-z
ordering eshell-up
ordering eros
ordering engine-mode
ordering emojify
ordering emms
ordering ellocate
ordering elisp-refs
ordering elisp-demos
ordering elfeed-score
ordering elfeed-org
ordering elfeed
ordering el-mock
ordering el-get
ordering edit-indirect
ordering doct
ordering dmenu
ordering dbc
ordering db
ordering dashboard
ordering dash-functional
ordering dash
ordering counsel
ordering corfu
ordering consult
ordering bind-key
ordering avy
ordering auto-capitalize
ordering auth-source-pass
ordering async
ordering annalist
ordering anaphora
ordering all-the-icons
ordering aggressive-indent
ordering aggressive-fill-paragraph
ordering ace-window
ordering org-super-links
ordering dirvish-media
ordering dirvish-icons
ordering dirvish-collapse
ordering dirvish-subtree
ordering dirvish
ordering search-web
ordering password-generator
ordering redacted
ordering tempel
ordering ednc
ordering with-emacs
ordering svg-lib
ordering svg-tag-mode
ordering iedit
ordering all-the-icons-completion
ordering buffer-expose
ordering embark
ordering default-text-scale
ordering evil-cleverparens
ordering zone-matrix
ordering zone-sl
ordering zone-rainbow
ordering iedit
ordering ef-themes
ordering spacemacs-theme
ordering decide
ordering evil-collection
ordering evil-goggles
ordering paredit
ordering elpaca
BEGINNING TO PROCESS QUEUES...
For information about GNU Emacs and the GNU system, type C-h C-a.
Mark set
Making completion list...
You can run the command ‘switch-to-buffer’ with C-x b
previous-line: Beginning of buffer [33 times]
V is undefined
Mark set
progfolio commented 1 year ago

Thanks for all the info. That makes it much easier to debug. I think what is actually going wrong is the combination of :depth 1 and a :ref in you recipes. You helped me fix a bug in the process which checks out the ref as well (there was no filter function, which was why we weren't getting a good error logged in this case).

So now there's the design question of how to handle this case. The simplest solution is to ignore :depth when :ref is used. That will result in a full repository clone which guarantees (so long as the ref is valid) the ref should be reachable. I've implemented this in the latest version of Elpaca.

Another approach would be to attempt to check out the ref at the specified depth, and if that fails, "git fetch --deepen X" until the ref is checked out or the repository history is exhausted. X could be a defcustom to allow the user to decide on the tradeoff between disk space and cloning speed.

And yet another approach, which relies on a new(ish) git fetch feature:

https://stackoverflow.com/questions/31278902/how-to-shallow-clone-a-specific-commit-with-depth-1

I'm hesitant to implement that because apparently the server and the client have to be running a version capable of fetching with a commit in addition to a "--depth".

A couple of notes about your config:

I would avoid the emacs-straight repository unless you need it. Its build script doesn't always get things correct.

I was able to get all of your recipes to build spare dirvish, which failed with:

fatal: reference is not a tree: be6f1320c47c5d48a5ba32c4e7dd4999bd55e750

And a local clone of https://github.com/alexluigit/dirvish does not seem to contain that commit hash:

 git log --reflog --pretty=tformat:"%H" | grep be6
 2f35e48d09b848be661c5f5734264330e824b0f3
a22769e1d338df1ff769c0be65d9a17700833cfb
d1dca1de3e6f95abe60d37121efc2dfe66ad3579
ed4ca897811d2955bbe6de3619821570fddb8b5e
caa0be6633ac7d76edd438fc9bfe39278b33e1cd
ffe29d991b6c9c027bc44d336787b93a3107be6a
0be1be6886b4321135e349814fff056934c309d6
e80a9e06c25ed22461ba223291be60f8b8829285

For your testing purposes:

(add-hook 'elpaca-after-init-hook (lambda (&rest _) "DONE INSTALLING PACAKGES..."))

You'll probably want to message that string:

(add-hook 'elpaca-after-init-hook (lambda (&rest _) (message "%s" "DONE INSTALLING PACAKGES...")))
Luis-Henriquez-Perez commented 1 year ago

Another approach would be to attempt to check out the ref at the specified depth, and if that fails, "git fetch --deepen X" until the ref is checked out or the repository history is exhausted. X could be a defcustom to allow the user to decide on the tradeoff between disk space and cloning speed.

I think I did something like this in my straight setup. I looked at this question for the idea.

(defun oo-ensure-minimal-depth (recipe)
  "Fetch from the repo until it's at TARGET-COMMIT."
  (when-let* ((package (car recipe))
          (recipe (cdr recipe))
          (local-repo (plist-get recipe :local-repo))
          (repo-dir (straight--repos-dir local-repo))
          (target-commit (plist-get recipe :commit))
          (current-commit (straight-vc-get-commit 'git local-repo))
          (depth (oo-depth repo-dir))
          (last-depth (1- depth)))
    (message "Finding minimal depth for %s (%s -> %s)..."
         package
         (seq-take current-commit 6)
         (seq-take target-commit 6))
    (while (and (not (= last-depth depth))
            (not (straight-vc-commit-present-p recipe target-commit)))
      (oo-call-process "git" "-C" repo-dir "fetch" (format "--depth=%s" (1+ depth)))
      (setq current-commit (straight-vc-get-commit 'git local-repo))
      (setq last-depth depth)
      (setq depth (oo-depth repo-dir)))
    (when (= last-depth depth) (message "Can't keep diving."))
    (when (straight-vc-commit-present-p recipe target-commit)
      (message "Done diving. Target reached."))
    (message "Set depth of %s to %s." package depth)))
Luis-Henriquez-Perez commented 1 year ago

I like your second approach the most. For me the extra time installing packages (which should only be a one time cost) is worth saving disk space by not getting full clones.

Thanks for the notes on my config.

I would avoid the emacs-straight repository unless you need it.

I have removed the references to "emacs-straight", replacing them with the original repos.

And a local clone of https://github.com/alexluigit/dirvish does not seem to contain that commit hash:

This was probably a mistake by me. I changed the commit hash.

You'll probably want to message that string:

You're right, nice catch!

regarding upgrading elpaca and its effect on other recipes

I have a concern with recipes. Right now the way I have them they are being merged with elpaca's built in recipes which is good because I know the commit hash and the repo I want but the default recipes might have some complex :files property that I don't understand/know about. So it's a good combination from my knowledge and elpaca's knowledge to come up with a good recipe. However, if I upgrade elpaca and it changes these unseen details, then my other recipes won't be exactly the same anymore. That's why I think I'll have to use something like elpaca-recipe to replace my recipes with the full resulting recipe. This way even if I upgrade elpaca other recipes will remain exactly the same.

ensuring I have a recipe for every package and dependency

I am concerned that I install a package which has a dependency I wasn't aware of and then I won't save the recipe for that dependency in my list of recipes. Maybe the lockfile which I see is a planned feature will help with this. Maybe instead of like in straight were the lockfile was an alist of (package-name . commit-hash), the lockfile for elpaca should just be a list of full recipes for every package installed with elpaca. It didn't make sense to me anyway to have both a list of recipes and a lockfile with (package-name . commit-hash)--especially in elpaca's case where the recipe can contain the commit hash.

orphans or unneeded packages

I'd like to know if there are some packages that I no longer need anymore. Will removing a packages also remove its unneeded dependencies as well? I just wanted to jot this thought down. As I start using elpaca I'll see how this will work in practice.

progfolio commented 1 year ago

That's why I think I'll have to use something like elpaca-recipe to replace my recipes with the full resulting recipe.

Maybe instead of like in straight were the lockfile was an alist of (package-name . commit-hash), the lockfile for elpaca should just be a list of full recipes for every package installed with elpaca.

Lockfiles should address this concern. Still a work in progress. The elpaca-write-lockfile command does exactly what you're describing. The full recipe is recorded along with a commit :ref.

elpaca-load-lockfile is stubbed out, but not functional yet. I still have some design considerations to think about.

I'd like to know if there are some packages that I no longer need anymore.

m-x elpaca-manager then O, which is bound to elpaca-ui-search-orphaned. This will show packages which have $ELPACA/repos or $ELPACA/builds directories on disk, but were not declared during init or installed during a session via elpaca-try.

I'd like to know if there are some packages that I no longer need anymore.

In addition to the elpaca-ui-search-orphaned command mentioned above, T is bound to elpaca-ui-search-tried, which will show packages you've installed during the current Emacs session. Such packages will become orphans the next time Emacs is started unless you add them to your init.

Sometimes I keep a few "orphaned" packages around when I'm experimenting with or contributing to packages. If you re-install them, it should be nearly instantaneous due to everything already being built on disk. Just something to keep in mind.

Will removing a packages also remove its unneeded dependencies as well?

Via the UI, yes. Dependencies are deleted unless another package depends on them. The elpaca-delete command needs some reworking to its interactive spec to allow that via a prefix-arg (I'll fix that soon), but it is possible programmatically now.

I think we've addressed the initial concern of this issue, so I'm going to close it to keep things organized. If you run into any other problems or have any other questions please feel free to open more issues.