radian-software / straight.el

🍀 Next-generation, purely functional package manager for the Emacs hacker.
MIT License
2.71k stars 150 forks source link

Discussion: using package.el and straight.el together #128

Closed rgrinberg closed 6 years ago

rgrinberg commented 7 years ago

straight.el looks like a real awesome package manager and I'd like to start using it right away. But I already have a massive spacemacs based config, so I'm wondering if I can make a gradual change towards straight somehow. For example, only using it to manage a few packages to start. Wondering if this is a good idea and is possible at all.

raxod502 commented 7 years ago

Well, I would say that straight.el is much better than package.el in this respect, since straight.el will do only what you explicitly tell it to do in your init-file, and nothing more. As opposed to package.el which insists on activating all the packages you ever installed with it.

There should be no trouble at all with using both straight.el and other package managers, at least from the straight.el side. If you are going to be using straight.el together with package.el, you may want to set straight-enable-package-integration to nil, since otherwise straight.el will automatically disable some "features" (i.e. bugs which are defended vehemently on the mailing list) of package.el that cause package.el to modify the init-file.

The thing you have to be concerned about is duplicate packages. There is no inherent harm in loading a package twice, other than that your init may be slower and you might get some weird side effects if the two versions being loaded are not the same. You should be especially concerned about dependencies, since if you use package manager X to load package Y, then you also need to use package manager X to load all of the dependencies of package Y—otherwise you'll get duplicate packages if you load those dependencies also using package manager Z.

Another consideration is that use-package does not have support for using different package managers for the :ensure clauses in different use-package forms. This is not a problem straight.el can fix, although you may want to keep an eye on my new with-feature package, which will solve this problem among others. As a workaround, I would recommend disabling :ensure for the use-package forms that you wish to use straight.el, and then just putting a manual (straight-use-package 'whatever) above those use-package forms. I.e.

(straight-use-package 'el-patch)
(use-package el-patch
  :ensure nil
  <whatever>)

instead of

(use-package el-patch
  :ensure t
  <whatever>

In order for the use-package forms that do specify :ensure to keep using package.el rather than straight.el, you should set straight-enable-use-package-integration to nil.

Does this answer your question? Let me know if any other information would be helpful.

raxod502 commented 7 years ago

Also, for the question of "is it safe to use" in general, see the new known issue FAQ section.

caadar commented 7 years ago

Hi, what is "different use-package forms" you mentioned?

Are you recommend to use such style

(straight-use-package 'el-patch)
(use-package el-patch
 :ensure nil
 <whatever>)

for every package installed?

Currently I set config with

(use-package el-patch
 :ensure t
 <whatever>)

only. It seems works.

vyp commented 7 years ago

@caadar No, that's just for @rgrinberg's particular use case. What you're doing is already recommended. 😀

raxod502 commented 7 years ago

that's just for @rgrinberg's particular use case

Correct, the problem only arises when you want to use straight.el for some packages and another package manager for others.

rgrinberg commented 7 years ago

@raxod502 thanks for the reassuring response. I'm trying out straight.el and things are going great except for one situation. There are some packages where I'd like straight to simply install them from an existing local repositories instead of trying to clone them. Those are usually packages that aren't just emacs packages and hence need to be located where they are in other (non emacs) contexts. For example, the utop package that I'm trying to get working:

(straight-use-package
 '(utop
   :type git :repo "rgrinberg/utop"
   :local-repo "/home/rgrinberg/reps/ml/utop"
   :files "src/top/utop.el"
   :upstream (:host github :repo "diml/utop")))

Ideally, straight-el should simply assume that this package is cloned and just build it. But I can't figure out how to do that.

raxod502 commented 7 years ago

Currently straight.el only supports local repositories that are inside ~/.emacs.d/straight/repos. That is, :local-repo isn't resolved as a real path, it's just concatenated to the prefix I mentioned. So that's your problem.

The quick fix is to symlink the repository into ~/.emacs.d/straight/repos. In general, I would consider allowing a :local-repo that is outside of your ~/.emacs.d directory, if that would be a useful feature.

rgrinberg commented 7 years ago

Thanks, this workaround is adequate. I think straight supporting local repos anywhere on the filesystem would be a useful feature.

raxod502 commented 7 years ago

I've created https://github.com/raxod502/straight.el/issues/133 to track this feature separately. Feel free to either close this issue or follow up with other questions.

rgrinberg commented 7 years ago

Thanks again :) I will create new issues/questions as necessary

rgrinberg commented 7 years ago

Actually, I have another question that probably doesn't deserve its own issue:

Is there a way to prevent straight from hacking use-package? The reason for that is that I already have use-package calls done by my spacemacs config and those packages are already being installed by the traditional package management system. It would be preferable would only install packages that are used through straight-use-package only.

raxod502 commented 7 years ago
(setq straight-enable-use-package-integration nil)

for more info, Integration with use-package. That's what I advise in https://github.com/raxod502/straight.el/issues/128#issuecomment-321956872, to allow for the default to remain package.el but for some packages to be installed via straight.el.

rgrinberg commented 7 years ago

OK, thanks.

raxod502 commented 7 years ago

… except for the part where straight-enable-use-package-integration doesn't actually do anything(?!!?!), not sure how I screwed that up.

raxod502 commented 7 years ago

Fixed on develop.

emacs18 commented 7 years ago

@raxod502 said "package.el which insists on activating all the packages you ever installed with it."

I would like to point out that this is not necessarily the case if you call package-initialize with non-nil optional argument, e.g.,

(package-initialize 'no-activate)

This is how I have always use package.el so that I can have many installed packages, but activate only some of them by using a different init file when emacs starts.

raxod502 commented 7 years ago

@emacs18 You're right. It's only the default behavior which is flawed, at least in this case.

If you have a suggestion for a corresponding update to the comparisons section in the README, I'd be glad to consider it.

emacs18 commented 7 years ago

@raxod502 I'll look into providing a patch later.

In the mean time, I would like to ask you if the following could allow me to use straight.el along with spacemacs without duplicating packages:

(defun my--advice-package-installed-p (original-function &rest args)
  "Return t if package is installed via `straight' package manager."
  (or (and (fboundp 'straight--dir)
           (straight--dir "build" (symbol-name (car args))))
      (apply original-function args)))

(advice-add 'package-installed-p :around 'my--advice-package-installed-p)

I'm in the process of testing this idea. I also considered writing use-package-ensure-straight similar to use-package-ensure-elpa to be set as the value of use-package-ensure-function variable. However just advising package-installed-p seemed like a simpler idea.

raxod502 commented 7 years ago

@emacs18 Your advice doesn't take into account the case where a package was previously loaded using straight.el, but is not currently. Use this instead:

(defun my--advice-package-installed-p (original-function &rest args)
  "Return t if package is installed via `straight' package manager."
  (or (gethash straight--recipe-cache (car args))
      (apply original-function args)))

(advice-add 'package-installed-p :around 'my--advice-package-installed-p)

At some point it may be a good idea to expose some user-facing functions for getting information from the caches.

That said I can't guarantee that advising package-installed-p will actually stop package.el from loading duplicate packages. In fact, if I had to guess, I would say that duplicate loading will still take place.

By the way, straight.el already provides straight-use-package-ensure-function which it sets as the value of use-package-ensure-function. You would of course have to override this with your own version that delegates to both straight.el and package.el conditionally.

raxod502 commented 7 years ago

Actually, your original advice doesn't work at all. It always returns t.

emacs18 commented 7 years ago

Yes I forgot to add (file-directory-p) call. Thanks for your advice. I'll poke around some more.

rgrinberg commented 7 years ago

The way :upstream works doesn't seem to work for a use case that I have. My standard practice for naming remotes is as follows:

Where I only pull from origin and only push to rgrinberg. Is there a way to make straight respect this convention? Since I do have some emacs package repositories that live outside of straights control.

raxod502 commented 7 years ago

@rgrinberg Currently, there is no way to specify different remote names on a per-repository basis. However, you can change them globally by configuring straight-vc-git-primary-remote and straight-vc-git-upstream-remote. That probably doesn't really work for your use case, since you sometimes want origin to be the primary remote, and sometimes the :upstream (in the case that you have a fork). This specific problem will be alleviated by #114, but more generally it would be nice to allow customizing remote names directly in the recipe. I suggest you open a new issue to track it.

emacs18 commented 7 years ago

The code shown below allowed me to use straight.el and packge.el to install any number of packages using straight.el, but let package.el (via use-package) install the rest. My method of verifying that there were no duplications was to make sure that load-path did not have path added by both straight.el and package.el.

(defun my-straight-installed-p (package)
  "Return non-nil if PACKAGE is installed by `straight'."
  (and (fboundp 'straight--dir)
       ;; FIXME: Radon said to use this gethash call rather than straight--dir.
       ;; However this fails with "Wrong type argument: hash-table-p,
       ;; org-plus-contrib". So use straight--dir for now.
       '(gethash straight--recipe-cache package)
       ;; 'dash' package is cloned under 'dash.el' directory. That is why we
       ;; check for raw name as well as with ".el" appended.
       (or (file-directory-p (straight--dir "build" (symbol-name (car args))))
           (file-directory-p (concat (straight--dir "build" (symbol-name (car args))) ".el")))
       ))

(defun my--advice-package-installed-p (original-function &rest args)
  "Return t if package is installed via `straight' package manager. Otherwise
call the original function `package-installed-p'."
  (or (my-straight-installed-p (car args))
      (apply original-function args)))
(advice-add 'package-installed-p :around 'my--advice-package-installed-p)

(defun my--advice-package-activate (original-function &rest args)
  "Return t if package is installed via `straight' package manager. Otherwise
call the original function `package-activate'."
  (if (my-straight-installed-p (car args))
      (progn
        (unless (memq (car args) package-activated-list)
          ;; Not sure if package-activated-list needs to be updated here ...
          (push (car args) package-activated-list))
        t)
    (apply original-function args)))
(advice-add 'package-activate :around 'my--advice-package-activate)

With this method I was able to install any number packages by straight.el, and the rest by package.el. At first I started with less than 10 packages installed by straight.el, and 310+ installed by package.el. I then gradually added more packages to be installed by straight.el. Now I have pretty much all 320+ pacakges installed via straight.el and only a handful via package.el.

Spacemacs came up fine as well.

However calling straight-use-package on about 320 packages took more than 14 minutes. This is on a virtual ubuntu running via virtualbox with win10 as host OS. Spacemacs on this same setup comes up in about 6 seconds if all 320+ packages are setup via package.el without using straight.el.

If the startup time can be reduced from about 15 minutes to less than a minute, then I might be motivated to switch over to using straight.el exclusively and not use package.el at all. I'm hoping that the slow startup is due to my mistake which can easil be fixed.

Following is my init file to test straight.el. Code to setup spacemacs and my other customization (includeing the code shown above) followed the code below. This is not shown since it is not relevant.

;; (package-initialize)

(setq my-start-time (current-time))

(defun my-print-time (msg)
  (let ((elapsed (float-time (time-subtract (current-time) my-start-time))))
    (message "%s (%.3fs)" msg elapsed)))

(setq user-emacs-directory "/u/me/elisp/")

(custom-set-variables '(straight-enable-package-integration nil))

(load "/u/me/Public/straight.el/straight")

(straight--reset-caches)

(unless (and after-init-time (not (bound-and-true-p straight-treat-as-init)))
  (add-hook 'after-init-hook #'straight-finalize-transaction)
  (straight-begin-transaction)
  (straight-mark-transaction-as-init))

;; So meta. This updates the various caches, so that straight.el shows
;; up properly in the lockfile and other things like that.
(straight-use-package '(straight :type git :host github
                                 :repo "raxod502/straight.el"
                                 :files ("straight.el")))

;; Now we need to register the default recipe repositories.

(straight-use-recipes '(melpa :type git :host github
                              :repo "melpa/melpa"
                              :no-build t))

(straight-use-recipes '(emacsmirror :type git :host github
                                    :repo "emacsmirror/epkgs"
                                    :nonrecursive t
                                    :no-build t))

(mapcar
 #'straight-use-package
 '(magit org quelpa use-package))

(my-print-time "straight-use-package 4 pkgs")

(mapcar
 #'straight-use-package
 '(
   ac-ispell ace-jump-helm-line ace-link ace-window adaptive-wrap ag
   aggressive-indent alert anaconda-mode anaphora anzu apiwrap async auctex
   auto-compile auto-complete auto-dictionary auto-highlight-symbol
   auto-yasnippet avy bbdb beacon beacon biblio biblio-core bind-key bind-map
   browse-at-remote clang-format clean-aindent-mode clojure-mode closql
   cmake-mode coffee-mode column-enforce-mode command-log-mode company company
   company-anaconda company-anaconda company-auctex company-c-headers
   company-c-headers company-math company-math company-math company-quickhelp
   company-quickhelp company-shell company-shell company-statistics
   company-statistics company-tern company-tern company-web company-web counsel
   counsel-dash counsel-projectile crux cython-mode dash dash-functional
   default-text-scale deferred define-word demangle-mode diff-hl diminish
   disable-mouse disaster dumb-jump edit-server editorconfig ein elfeed
   elisp-slime-nav elpa-mirror elpy emacsql emacsql-sqlite emmet-mode emms
   engine-mode epkg epl esh-help eshell-prompt-extras eshell-z eval-sexp-fu evil
   evil-anzu evil-args evil-ediff evil-escape evil-exchange evil-iedit-state
   evil-indent-plus evil-lion evil-lisp-state evil-magit evil-matchit evil-mc
   evil-nerd-commenter evil-numbers evil-org evil-search-highlight-persist
   evil-surround evil-tutor evil-visual-mark-mode evil-visualstar
   exec-path-from-shell expand-region eyebrowse f fancy-battery
   fill-column-indicator finalize find-file-in-project fish-mode floobits flx
   flx-ido flycheck flycheck-bashate flycheck-pos-tip flyspell-correct
   flyspell-correct-ivy fringe-helper fuzzy geiser ggtags gh-md ghub ghub+
   git-commit git-gutter git-gutter+ git-gutter-fringe git-gutter-fringe+
   git-link git-messenger git-timemachine gitattributes-mode gitconfig-mode
   gitignore-mode gntp gnuplot golden-ratio google-c-style google-translate
   goto-chg graphviz-dot-mode haml-mode helm helm-ag helm-bibtex
   helm-c-yasnippet helm-company helm-core helm-css-scss helm-dash
   helm-descbinds helm-dictionary helm-flx helm-gitignore helm-gtags helm-make
   helm-pydoc help-fns+ hide-comnt highlight highlight-indentation
   highlight-numbers highlight-parentheses highlight-thing history hl-todo ht
   htmlize hungry-delete hy-mode hydra ibuffer-projectile iedit igrep imenu-list
   impatient-mode indent-guide info+ insert-shebang irony itail ivy ivy-hydra
   ivy-purpose js-doc js2-mode js2-refactor json-mode json-reformat
   json-snatcher key-chord less-css-mode link-hint linum-relative lispy
   live-py-mode livid-mode load-relative loc-changes log4e macrostep
   magit-gitflow magit-popup magithub markdown-mode markdown-toc
   math-symbol-lists mmm-mode mmt monokai-theme move-text multi-term
   multiple-cursors neotree ob-ipython open-junk-file org-alert org-brain
   org-bullets org-download org-mime org-password-manager org-pomodoro
   org-present org-projectile org-tree-slide ox-gfm ox-pandoc ox-reveal p4
   package-build packed pacmacs paradox paredit parent-mode parsebib
   password-generator password-store pcmpl-git pcre2el persp-mode
   pip-requirements pkg-info popup popwin pos-tip powerline projectile pug-mode
   py-isort pyenv-mode pytest pythonic pyvenv rainbow-delimiters realgud request
   request-deferred restart-emacs s sass-mode scss-mode shell-pop showkey
   simple-call-tree simple-httpd skewer-mode slim-mode smartparens smeargle smex
   spaceline spinner string-inflection sudoku swiper symon tagedit tern
   test-simple tldr toc-org typit undo-tree uuidgen vi-tilde-fringe
   volatile-highlights web-beautify web-completion-data web-mode websocket wgrep
   wgrep-ag which-key window-purpose winum with-editor worf ws-butler xah-lookup
   xterm-color yapfify yasnippet zeal-at-point zoutline
   ))

(my-print-time "straight-use-package many pkgs")
emacs18 commented 7 years ago

I tried to profile via the following:

(elp-instrument-package "straight")
(elp-reset-all)
(mapcar #'straight-use-package '( ... )) ;; about 170 pacakges
(elp-results)

Following shows the profiling result. Not sure if this is helpful. Please let me know if there is anything else I can do to help understand what is taking so long.

straight--transaction-exec                     2253        700.71348628  0.3110135314
straight-use-package                           1634        355.84875113  0.2177776934
straight--package-might-be-modified-p          467         347.24724488  0.7435701175
straight--cache-package-modifications          171         347.22685301  2.0305663919
straight--convert-recipe                       2440        1.8821768900  0.0007713839
straight-recipes-retrieve                      485         1.8590368450  0.0038330656
straight-recipes                               806         1.8564849809  0.0023033312
straight-recipes-emacsmirror-retrieve          321         1.7303635509  0.0053905406
straight-finalize-transaction                  1634        1.2360444370  0.0007564531
straight--save-build-cache                     171         1.2247415299  0.0071622311
straight--load-build-cache                     171         0.7969233699  0.0046603705
straight--activate-package-autoloads           467         0.5217148469  0.0011171624
straight-recipes-melpa-retrieve                485         0.0454132690  9.363...e-05
straight--get-call                             321         0.032518909   0.0001013050
straight--get-call-raw                         321         0.032428036   0.0001010219
straight-vc                                    830         0.0314635299  3.790...e-05
straight--register-recipe                      748         0.0314228160  4.200...e-05
straight-vc-keywords                           660         0.0275945299  4.180...e-05
straight--dir                                  3004        0.0262487459  8.737...e-06
straight--file                                 809         0.0212868049  2.631...e-05
straight--repository-is-available-p            748         0.018435396   2.464...e-05
straight--uncache-package-modifications        171         0.0093695809  5.479...e-05
straight-begin-transaction                     1634        0.0090953039  5.566...e-06
straight--add-package-to-load-path             467         0.0071420979  1.529...e-05
straight-vc-local-repo-name                    170         0.0050447479  2.967...e-05
straight-vc-git-local-repo-name                170         0.0027370300  1.610...e-05
straight--get-overridden-recipe                1634        0.0018399910  1.126...e-06
straight--transaction-p                        2253        0.0009042939  4.013...e-07
straight--autoload-file-name                   467         0.0006129540  1.312...e-06
straight--checkhash                            467         0.0005662019  1.212...e-06
straight--get-dependencies                     467         0.0005402970  1.156...e-06
straight-vc-git-keywords                       660         0.0002679830  4.060...e-07
straight-vc-git--decode-url                    6           3.3597e-05    5.599...e-06
raxod502 commented 7 years ago
;; FIXME: Radon said to use this gethash call rather than straight--dir.
;; However this fails with "Wrong type argument: hash-table-p,
;; org-plus-contrib". So use straight--dir for now.

Lol that's just because I swapped the arguments to gethash. Use

(gethash (car args) straight--recipe-cache)

instead.

;; 'dash' package is cloned under 'dash.el' directory. That is why we
;; check for raw name as well as with ".el" appended.

This shouldn't be necessary if you are checking the build directory, which uses package names rather than local repository names. You would need this if you were checking the repos directory instead, although in that case the correct approach would be to check the :local-repo attribute on the recipe in straight--recipe-cache. In any case this is superfluous since you can just use the fixed gethash command above.

(load "/u/me/Public/straight.el/straight")

I am really confused about why you are doing all this manually instead of using the bootstrap snippet. The latter should have the same effect.

All this said, even though straight.el has significant performance problems (see #9, #118, #119, #129), I only expect loading 320 packages to take 5.8s (extrapolating linearly from my own config). 15 minutes is just ridiculous; there must be something terribly wrong. Let me see if I can reproduce the behavior you describe using your code.

raxod502 commented 7 years ago

OK. With the following minified init.el:

(setq package-enable-at-startup nil)
(load (concat user-emacs-directory "straight/bootstrap.el") nil 'nomessage)

(mapcar
 #'straight-use-package
 '(
   ac-ispell ace-jump-helm-line ace-link ace-window adaptive-wrap ag
   aggressive-indent alert anaconda-mode anaphora anzu apiwrap async auctex
   auto-compile auto-complete auto-dictionary auto-highlight-symbol
   auto-yasnippet avy bbdb beacon beacon biblio biblio-core bind-key bind-map
   browse-at-remote clang-format clean-aindent-mode clojure-mode closql
   cmake-mode coffee-mode column-enforce-mode command-log-mode company company
   company-anaconda company-anaconda company-auctex company-c-headers
   company-c-headers company-math company-math company-math company-quickhelp
   company-quickhelp company-shell company-shell company-statistics
   company-statistics company-tern company-tern company-web company-web counsel
   counsel-dash counsel-projectile crux cython-mode dash dash-functional
   default-text-scale deferred define-word demangle-mode diff-hl diminish
   disable-mouse disaster dumb-jump edit-server editorconfig ein elfeed
   elisp-slime-nav elpa-mirror elpy emacsql emacsql-sqlite emmet-mode emms
   engine-mode epkg epl esh-help eshell-prompt-extras eshell-z eval-sexp-fu evil
   evil-anzu evil-args evil-ediff evil-escape evil-exchange evil-iedit-state
   evil-indent-plus evil-lion evil-lisp-state evil-magit evil-matchit evil-mc
   evil-nerd-commenter evil-numbers evil-org evil-search-highlight-persist
   evil-surround evil-tutor evil-visual-mark-mode evil-visualstar
   exec-path-from-shell expand-region eyebrowse f fancy-battery
   fill-column-indicator finalize find-file-in-project fish-mode floobits flx
   flx-ido flycheck flycheck-bashate flycheck-pos-tip flyspell-correct
   flyspell-correct-ivy fringe-helper fuzzy geiser ggtags gh-md ghub ghub+
   git-commit git-gutter git-gutter+ git-gutter-fringe git-gutter-fringe+
   git-link git-messenger git-timemachine gitattributes-mode gitconfig-mode
   gitignore-mode gntp gnuplot golden-ratio google-c-style google-translate
   goto-chg graphviz-dot-mode haml-mode helm helm-ag helm-bibtex
   helm-c-yasnippet helm-company helm-core helm-css-scss helm-dash
   helm-descbinds helm-dictionary helm-flx helm-gitignore helm-gtags helm-make
   helm-pydoc help-fns+ hide-comnt highlight highlight-indentation
   highlight-numbers highlight-parentheses highlight-thing history hl-todo ht
   htmlize hungry-delete hy-mode hydra ibuffer-projectile iedit igrep imenu-list
   impatient-mode indent-guide info+ insert-shebang irony itail ivy ivy-hydra
   ivy-purpose js-doc js2-mode js2-refactor json-mode json-reformat
   json-snatcher key-chord less-css-mode link-hint linum-relative lispy
   live-py-mode livid-mode load-relative loc-changes log4e macrostep magit
   magit-gitflow magit-popup magithub markdown-mode markdown-toc
   math-symbol-lists mmm-mode mmt monokai-theme move-text multi-term
   multiple-cursors neotree ob-ipython open-junk-file org org-alert org-brain
   org-bullets org-download org-mime org-password-manager org-pomodoro
   org-present org-projectile org-tree-slide ox-gfm ox-pandoc ox-reveal p4
   package-build packed pacmacs paradox paredit parent-mode parsebib
   password-generator password-store pcmpl-git pcre2el persp-mode
   pip-requirements pkg-info popup popwin pos-tip powerline projectile pug-mode
   py-isort pyenv-mode pytest pythonic pyvenv quelpa rainbow-delimiters realgud
   request request-deferred restart-emacs s sass-mode scss-mode shell-pop showkey
   simple-call-tree simple-httpd skewer-mode slim-mode smartparens smeargle smex
   spaceline spinner string-inflection sudoku swiper symon tagedit tern
   test-simple tldr toc-org typit undo-tree use-package uuidgen vi-tilde-fringe
   volatile-highlights web-beautify web-completion-data web-mode websocket wgrep
   wgrep-ag which-key window-purpose winum with-editor worf ws-butler xah-lookup
   xterm-color yapfify yasnippet zeal-at-point zoutline
   ))

I observe an M-x emacs-init-time of 3.6 seconds. Can you test with this version @emacs18?

raxod502 commented 7 years ago

Note, by the way: you will observe very very poor performance of straight.el in the case that you evaluate a lot of straight-use-package calls without wrapping said evaluation in a transaction. By default, init is considered a transaction. You are encouraged to also insert straight-transaction blocks into any of your interactive commands that evaluate Elisp code interactively.

In particular, your profiling

(elp-instrument-package "straight")
(elp-reset-all)
(mapcar #'straight-use-package '( ... )) ;; about 170 pacakges
(elp-results)

will have terrible performance compared to if you wrap the mapcar in a straight-transaction. This is due to several reasons, but the biggest is that straight.el checks all packages for modifications at the beginning of a transaction, because this is faster than checking each package individually. So if you don't wrap in a transaction, then you're checking all 320 packages for modifications 320 times. O(n²) makes everybody cry.

Addendum: if you look in your profiling results, it's clear that straight--cache-package-modifications is the bottleneck, as predicted by this analysis.

Addendum 2: any questions about the transaction system? Or suggestions on how I can make the important considerations more obvious in the documentation?

raxod502 commented 7 years ago

Also, food for thought. Would anybody be interested in me making a pull request to Spacemacs to add an option for using straight.el instead of package.el (much like there is an option to use helm vs. ivy or evil vs. the One True Keybindings)?

vyp commented 7 years ago

@raxod502 This thread may interest you in that case (or maybe not :woman_shrugging:): https://github.com/quelpa/quelpa/issues/93#issuecomment-143230503

raxod502 commented 7 years ago

Created https://github.com/syl20bnr/spacemacs/issues/9505.

emacs18 commented 7 years ago

Thanks for all the feedback. I found out why things were taking so long. It was because I launched emacs via

emacs -Q -l .../init-file-for-straight.el

If I do not use -Q and setup my ~/.emacs to do the equivalent thing, then the startup time goes from 15 minutes down to 4 seconds. Now that I know this, I'll just avoid using -Q, and instead mess with my ~/.emacs to setup straight.

raxod502 commented 7 years ago

See also this snippet from the user manual, which explains the proper way to load straight.el using emacs -Q -l .../init-file-for-straight.el.

There is one final user-facing note about the transaction system, which is important when you want to load your init-file after Emacs init has already completed, but before straight.el has been loaded (so you cannot just wrap the call in straight-transaction). To cover this edge case (which arises, for example, when you wish to profile your init-file using something like esup), you should use the following pattern:

(unwind-protect
    (let ((straight-treat-as-init t))
      "load your init-file here")
  (straight-finalize-transaction))
braham-snyder commented 6 years ago

@emacs18 any chance you want to share your most recent versions of my-straight-installed-p etc., and perhaps indicate where within your user-init-file / dotspacemacs-filepath you've put it all?

raxod502 commented 6 years ago

Closing this issue since we have a Gitter for general discussion now. For requesting a better-supported way to use straight.el and package.el together with use-package, open a new issue.

raxod502 commented 6 years ago

@rgrinberg Note that using package.el and straight.el together is now trivial with the new version of use-package, since straight.el uses the new :straight keyword while package.el continues to use the old :ensure keyword, and they can be combined.

bradyt commented 5 years ago

Does that mean straight-enable-use-package-integration should be deprecated?

raxod502 commented 5 years ago

I don't see why; can you explain?

bradyt commented 5 years ago

I thought most of the idea with straight-enable-use-package-integration, was to override the use of :ensure to use straight.el, instead of package.el. And later, I thought a separate keyword was added to use-package so that :ensure can now always correspond with package.el. Sorry if I misunderstand.

raxod502 commented 4 years ago

Everything you said is correct, except that you are omitting the part where in the default (i.e. non-deprecated) configuration, straight-enable-use-package-integration defines the :straight keyword that allows you to use straight.el with use-package. That part is definitely not deprecated as without it there is no way to use the packages together. You can read more in the docstrings of the relevant variables.