Dependencies missing from package

[Northcode]

In reference to: https://github.com/raxod502/straight.el/issues/233

Both outorg and outshine appear to be missing a bunch of dependency declarations. Outorg currently specifies: ;; Package-Requires: ((emacs "24.4")) However it actually requires: org,org-watchdoc, outline and emacs. Outshine currently specifies: ;; Package-Requires: ((outorg "2.0") (cl-lib "0.5")) It actually requires: cl, cl-lib, emacs, imenu, navi-mode, org, outline & outorg

This causes breakage if they are not installed or outorg is loaded before any of its dependencies not specified in the package, as it may load an incorrect version (like using built-in org instead of org installed from git or melpa) or just break because they arent installed.

[alphapapa]

Hi,

Well, some of those required files are actually included with Emacs and don't need to be declared as package dependencies. And some of the ones that Outshine requires are loaded by Outorg, which Outshine loads and declares a dependency on.

Specifically, the only package not included with Emacs that is loaded is navi-mode, and it is optional, so not a dependency. org-watchdoc also appears to be optional, and I can't even find it on my system.

Admittedly, it is a bit messy, and probably the best thing to do is a general refactoring of all 3 packages. I might do that someday, but no promises, as these are large packages that I didn't write myself.

However, the packages do install and load correctly when installed from MELPA with package.el. I think the problem you're having may be caused by straight.el being more strict than package.el.

The problem of using an outdated, built-in Org instead of a later version is a more general problem with Emacs package systems, AFAIK. But it shouldn't affect Outorg or Outshine, as they don't require recent Org versions.

Since I'm pretty sure this is not a bug on this end, I'm going to close this issue, but feel free to continue the discussion or reopen if necessary.

[raxod502]

I strongly believe this a bug in outorg/outshine. Just because a package is built-in doesn't mean you don't have to declare a dependency on it. There's an explicit category of package called built-in. Part of the reason for this distinction is that it allows package.el to install a package from some package archive if the built-in one is not a high enough version.

So to summarize, yes it looks like it works with package.el, but it doesn't really and it's conceptually wrong. Emacs packages that depend on org universally declare a dependency on it in Package-Requires.

It shouldn't be necessary to do any refactoring to fix this problem. Just add org to the Package-Requires header.

[alphapapa]

@raxod502 Thanks for your insights. I'm certainly willing to add a dependency to the headers if that will fix something, but I'm not entirely clear on what the problematic behavior is that needs to be fixed here. I'm also not sure that declaring dependencies on built-in versions will make any difference, because as far as I know, these packages work fine on older Emacs versions, so I don't think there's any need to force certain versions of built-in packages.

@Northcode Can you give me some specific examples of errors you've encountered? If there are some specific errors or bugs that I'm not aware of, I'll be happy to fix them, but at the moment I don't see what specifically needs to be fixed. Thanks.

[Northcode]

Sure: here, steps I used to reproduce this was:

• Use the init file from the issue here https://github.com/raxod502/straight.el/issues/233
• run (org-version)
• try to use org-capture

Output:

Org-mode version 8.2.10 (release_8.2.10 @ /home/andreas/emacs-testing/.emacs.d/straight/build/org/)
Problems while trying to load feature ‘org-w3m’
Problems while trying to load feature ‘org-bbdb’
Problems while trying to load feature ‘org-bibtex’
Problems while trying to load feature ‘org-docview’
Problems while trying to load feature ‘org-gnus’
Problems while trying to load feature ‘org-info’
Problems while trying to load feature ‘org-irc’
Problems while trying to load feature ‘org-mhe’
Problems while trying to load feature ‘org-rmail’
Template key: 
(New file)
org-element--set-regexps: Symbol’s function definition is void: org-link-types

It looks like this happens because its loaded the built-in org first, when outshine (require)'s it, and when use-package loads the new one, it overlaps weirdly and breaks a lot of stuff. If I reorder the use-packages to load org mode first, this doesn't happen, because its loaded the new org instead of the built-in.

[raxod502]

I'm not entirely clear on what the problematic behavior is that needs to be fixed here

The specific problematic behavior (as mentioned by @Northcode) arises when you load this package using a package manager that doesn't happen to perform the particular optimization that package.el uses (namely activating all installed packages at the same time).

The reason such problematic behavior arises is that this package depends on the org package, but doesn't say so. Like I said, just because the org package happens to have a version shipped with Emacs doesn't make it any less of a legitimate package.

I'm also not sure that declaring dependencies on built-in versions will make any difference

Maybe it won't for package.el, but it most certainly does for other package managers.

I don't think there's any need to force certain versions of built-in packages

It seems unlikely to me that this package works with Org version 1.0. There must be some minimum version. You could just make the minimum version of Org whatever version was shipped with the minimum version of Emacs that you support; that would be easy enough to find out.

[alphapapa]

@Northcode

It looks like this happens because its loaded the built-in org first, when outshine (require)'s it, and when use-package loads the new one, it overlaps weirdly and breaks a lot of stuff.

If I reorder the use-packages to load org mode first, this doesn't happen, because its loaded the new org instead of the built-in.

use-package is a macro that expands into, e.g. this:

(if (not (require 'org-sticky-header nil 'noerror))
    (ignore (message (format "Cannot load %s" 'org-sticky-header))))

As you can see, it just calls require. So whether (require 'org) is called from (require 'outorg) or (use-package org), the same version of Org should be loaded.

Now you're also using (use-package org :straight t), which means that straight is doing whatever it does. In other words, if you used (use-package org), it would make no difference whether (use-package outorg) or (use-package org) were called first.

Therefore the issue is being caused by straight, not by outorg.

Now having said that, why don't you just adjust your init file to fix the problem? e.g. instead of what you currently have:

(use-package outshine  ;; outshine depends on both outline and org, but will load the built-in org instead of checking if we have a straight.el version downloaded
  :straight t
  :init
  (require 'outshine)
  (add-hook 'outline-minor-mode-hook 'outshine-hook-function))

(use-package org
  :straight t
  :bind
  (("C-c a" . org-agenda)
   ("C-c c" . org-capture)))

Do this:

(use-package org
  :straight t
  :bind (("C-c a" . org-agenda)
         ("C-c c" . org-capture))
  :config
  (use-package outshine
    :straight t
    :config
    (add-hook 'outline-minor-mode-hook 'outshine-hook-function)))

That will probably work fine without any changes to this package or straight. Or if you don't want to nest them, just put (use-package outshine) after (use-package org ...).

[alphapapa]

@raxod502

The specific problematic behavior (as mentioned by @Northcode) arises when you load this package using a package manager that doesn't happen to perform the particular optimization that package.el uses (namely activating all installed packages at the same time).

Does this mean that you don't intend straight to be compatible with package.el? It's your project, so I'm not telling you what to do. But in general, it doesn't seem like you can expect existing Emacs packages, which have been designed to be used with package.el from the beginning, to make changes just for straight. Even if I were to make such changes to these packages, it wouldn't fix all the others, so users would still have issues like this. It seems like a long and frustrating "war," and similar compatibility issues MELPA has faced illustrate how troublesome one is. Wouldn't it be best to emulate package.el's behavior so it would be "drop-in compatible"?

The reason such problematic behavior arises is that this package depends on the org package, but doesn't say so. Like I said, just because the org package happens to have a version shipped with Emacs doesn't make it any less of a legitimate package.

In a way, yes; in a way, no. I don't think it's expected or required that packages declare dependencies on built-ins in the headers. As I said, even if I do so, it won't change all the other packages out there, so it seems like an issue that needs to be solved once-and-for-all at the package manager level rather than in individual packages.

I'm also not sure that declaring dependencies on built-in versions will make any difference

Maybe it won't for package.el, but it most certainly does for other package managers.

I don't know how other package managers behave in this context. I'm aware of package.el, el-get, borg, straight, and one or two others that I can't recall which seem to be mainly one-developer, one-user projects. IMO, alternative package managers should aim to be compatible with package.el rather than asking packages to change for them, even if this means "bug-for-bug compatibility" with package.el. (I realize that it's easy for me to say this, sitting here while you're actually writing an Emacs package manager, but the point remains.) Expecting the thousands of Emacs packages out there to change for you is a losing proposition.

I don't think there's any need to force certain versions of built-in packages

It seems unlikely to me that this package works with Org version 1.0. There must be some minimum version. You could just make the minimum version of Org whatever version was shipped with the minimum version of Emacs that you support; that would be easy enough to find out.

Of course I didn't mean Org 1.0. What I mean is, if the Emacs version is supported, then so is the built-in Org version included with that version of Emacs. AFAIK it's not expected or required to also declare a dependency on that specific built-in version, so if an alternative package manager doesn't work correctly without doing so, it seems like a bug in that package manager.

I think the bottom line is, these packages have thousands of downloads over the years, and therefore probably hundreds or thousands of active users. Now some cool, new, alternative package managers are being developed, which is great! But if they don't work with existing packages, that seems like a problem for these new package managers to fix, even if they are ultimately exposing buggy or undefined behavior in package.el. This is the burden of backward compatibility. It can't fairly or practically be pushed onto package authors.

[raxod502]

Does this mean that you don't intend straight to be compatible with package.el?

I regret only that GitHub does not provide me any larger font in which to say NO. Please refer to the Guiding principles section of the README, where the following is listed:

• No support whatsoever for package.el.

Perhaps I should put that bullet point in bold-face. Or <blink>, if only GitHub would support it in Markdown. Or both. package.el has fundamental design flaws which are inherently unfixable without a completely new approach. The point of straight.el is not to write a couple thousand lines of code which meticulously make exactly the same fundamental design flaws as package.el. If you like those design flaws, go use package.el instead. The point of straight.el is to fix the problems.

---

Does this mean that you don't intend straight to be compatible with package.el? It's your project, so I'm not telling you what to do. But in general, it doesn't seem like you can expect existing Emacs packages, which have been designed to be used with package.el from the beginning, to make changes just for straight. Even if I were to make such changes to these packages, it wouldn't fix all the others, so users would still have issues like this. It seems like a long and frustrating "war," and similar compatibility issues MELPA has faced illustrate how troublesome one is. Wouldn't it be best to emulate package.el's behavior so it would be "drop-in compatible"?

So, back in 2012 before package.el was standardized, I might say something like this to Tom Tromey:

Does this mean that you don't intend package.el to be compatible with manually loading packages in your init-file? It's your project, so I'm not telling you what to do. But in general, it doesn't seem like you can expect existing Emacs packages, which have been designed to be used the manual way from the beginning, to make changes just for package.el. Even if I were to make such changes to these packages, it wouldn't fix all the others, so users would still have issues like this. It seems like a long and frustrating "war." Wouldn't it be best to emulate the manual approach so it would be "drop-in compatible"?

And I think we are all very glad that despite the difficulty, Tom went ahead and pushed for a standardized package format. It was a Herculean effort, but well worth it.

Now if you look at what I'm proposing, it's not even remotely in the same ballpark in terms of complexity. In fact, I'm not even proposing anything new: the package.el specification already says you should treat built-in packages the same as external packages. The fact that you don't technically have to is purely a quirk of implementation, and indeed there are some circumstances even in package.el where failing to do so will mess things up.

I don't think it's expected or required that packages declare dependencies on built-ins in the headers.

I think this is just plain wrong. Take a look at what the Emacs Lisp Reference Manual has to say on the Package-Requires header. It gives the following as an example:

;; Package-Requires: ((gnus "1.0") (bubbles "2.7.2") cl-lib (seq))

So, their example header declares a dependency on the built-in cl-lib library. And I still maintain that this is expected by package.el, since cl-lib is also provided by GNU ELPA, and if a package requires a higher version of cl-lib than is shipped with Emacs, then package.el will download it from GNU ELPA. And if you're using a version of Emacs that doesn't ship cl-lib for some reason, then it would also download the package. But only if the dependency is properly declared.

it seems like an issue that needs to be solved once-and-for-all at the package manager level rather than in individual packages.

So... what is the solution you are proposing? Do I have straight.el grep package files to search for dependencies that aren't correctly declared? That is the very essence of a horrifying hack to work around a problem that has a much better solution.

And I'd like to add that this is the first package I've found that has this problem. Most packages do correctly declare dependencies on built-ins.

IMO, alternative package managers should aim to be compatible with package.el rather than asking packages to change for them, even if this means "bug-for-bug compatibility" with package.el.

straight.el has an immense amount of compatibility with package.el, in the sense that 99% of MELPA just works out of the box. But I am definitely not going to introduce bugs into straight.el for the sake of compatibility. That would be antithetical to the point of straight.el, which is to fix design flaws in package.el.

Expecting the thousands of Emacs packages out there to change for you is a losing proposition.

Sure, it would be—if I were expecting thousands of packages to change. I'm not, though. I'm expecting more like 10 packages to change. Yes, straight.el has exposed existing bugs in packaging, but not so many that the fixes are a burden.

AFAIK it's not expected or required to also declare a dependency on that specific built-in version

You're right. Instead of providing a version (org . "9.1.4") you can just list org in the Package-Requires header. I didn't actually realize you could do that until I checked the manual.

if an alternative package manager doesn't work correctly without doing so, it seems like a bug in that package manager.

Agreed as far as straight.el should not require specifying a version, since that's not a requirement in the package.el format. But since the package.el format does want you to specify dependencies on built-in packages, I don't think it's a bug in straight.el if the dependency handling is messed up as a result of such an omission.

if they don't work with existing packages, that seems like a problem for these new package managers to fix, even if they are ultimately exposing buggy or undefined behavior in package.el. This is the burden of backward compatibility. It can't fairly or practically be pushed onto package authors.

I believe you go too far. Sure, it's silly for a package manager to ask every package to change its format. But it is entirely reasonable for a package manager to ask 1% of packages to fix existing bugs that happen to manifest as bigger problems when using the new package manager.

[alphapapa]

I regret only that GitHub does not provide me any larger font in which to say NO. Please refer to the Guiding principles section of the README, where the following is listed:

Let's keep this friendly. :)

If you like those design flaws, go use package.el instead.

I am using package.el. :) I am definitely interested in straight.el and Borg, etc, but I'm waiting on them to mature before experimenting and possibly converting. Issues like this are an example of why. ;)

the package.el specification already says you should treat built-in packages the same as external packages

Can you point me to that specification? I'm far from an expert on the Emacs package systems, but I don't recall this requirement, and I don't think most packages do so in practice.

I think this is just plain wrong. Take a look at what the Emacs Lisp Reference Manual has to say on the Package-Requires header. It gives the following as an example:

With respect, I'm not sure that example qualifies as specifying that built-ins must be declared as dependencies. You may be right, but that may also be reading too much into a simple example that seems to be about syntax.

So, their example header declares a dependency on the built-in cl-lib library. And I still maintain that this is expected by package.el, since cl-lib is also provided by GNU ELPA, and if a package requires a higher version of cl-lib than is shipped with Emacs, then package.el will download it from GNU ELPA. And if you're using a version of Emacs that doesn't ship cl-lib for some reason, then it would also download the package. But only if the dependency is properly declared.

Again, that may be reading too much into it. Would a newer version of cl-lib even be expected to work on older Emacsen? I don't know the answer, but I would be surprised if it worked flawlessly. A newer version of cl-lib would likely use newer Emacs features, and since it's a built-in, I doubt it would be required to be backwards-compatible. e.g. loading cl-lib from Emacs 25.3 on Emacs 24.3 would probably not work.

So... what is the solution you are proposing?

I'm not developing a new package manager, nor am I using one, so I'm not proposing any solutions on your part. The suggestion I'm making for the user reporting this issue is to reorder the use-package clauses in his init file, which should solve the problem.

Do I have straight.el grep package files to search for dependencies that aren't correctly declared? That is the very essence of a horrifying hack to work around a problem that has a much better solution.

Well, in principle you're right, but in practice, grepping for (in rx syntax) (rx bol "(require" ... isn't that ugly of a hack. package-lint works that way, and it works well. But if you would prefer, a read/sexp-based solution would also work fine.

Now, what you do with the dependencies you parse out that way is another matter, but simply reading them from the file is straightforward enough. IMO if you want to be as robust and informative as possible, you should probably pursue that option, because otherwise you're depending on the package headers being correct and complete, and obviously that will not always be the case. :)

Sure, it would be—if I were expecting thousands of packages to change. I'm not, though. I'm expecting more like 10 packages to change.

Well, if there are only 10 packages on MELPA that don't work correctly when installed in infinite combinations and orders with straight.el, that would be great. Perhaps your faith is greater than mine. :) In any case, it's your project, and I'm not telling you how to run it.

You're right. Instead of providing a version (org . "9.1.4") you can just list org in the Package-Requires header. I didn't actually realize you could do that until I checked the manual.

Sure, but that won't fix the issue that @Northcode is reporting, because that wouldn't cause a newer, non-built-in version of Org to be downloaded. At least, it shouldn't, right?

Agreed as far as straight.el should not require specifying a version, since that's not a requirement in the package.el format. But since the package.el format does want you to specify dependencies on built-in packages, I don't think it's a bug in straight.el if the dependency handling is messed up as a result of such an omission.

Well, as I explained, in this case, IIUC, doing so would not fix this particular problem. (Assuming the problem is indeed that Org 8.x is loaded, then Org 9.x is loaded on top of it, which causes errors.)

I believe you go too far. Sure, it's silly for a package manager to ask every package to change its format. But it is entirely reasonable for a package manager to ask 1% of packages to fix existing bugs that happen to manifest as bigger problems when using the new package manager.

If they are actually bugs in the packages that violate the existing package.el specs, sure, it's reasonable--but IMO you're still fighting an uphill battle. e.g. a lot of packages are maintained by authors who rarely check bug reports, and MELPA has no non-maintainer-upload process like Debian does, so those problems are likely to remain for a long time...unless you're offering to take over the maintenance of those packages, and the authors are willing or long-term AFK, and MELPA agrees--so, are you? :)

This again raises the question of how many such packages there are, but I'm guessing it's not feasible to test the thousands of MELPA packages in infinite loading combinations and orders, especially when some of them do not declare dependencies on built-ins.

By the way, I am definitely no Steve Purcell, but I have reviewed probably 20-30 recipe pull requests on MELPA, from time-to-time, and IME they rarely, if ever, declare dependencies on built-ins. So I think this issue is much more common than you realize.

[raxod502]

Let's keep this friendly. :)

Yes, of course. Sorry, I intended my comment to be humorous. Just wanted to make a point :)

Can you point me to that specification?

I suppose specification is rather a strong word, but I was referring to the documentation of the package system in the Emacs Manual and Emacs Lisp Reference Manual. The question isn't addressed specifically, so is subject to interpretation, but in the absence of a distinction between built-in packages and non-built-in packages I would presume that they are to be treated equivalently.

Would a newer version of cl-lib even be expected to work on older Emacsen?

Answering that question is the purpose of the (emacs "21") dependency included in the Package-Requires header in cl-lib.el from GNU ELPA.

A newer version of cl-lib would likely use newer Emacs features, and since it's a built-in, I doubt it would be required to be backwards-compatible. e.g. loading cl-lib from Emacs 25.3 on Emacs 24.3 would probably not work.

That's true, but your comment only applies to the version shipped with Emacs, rather than the version provided on GNU ELPA. My comments concern when another version of a built-in package is also available from one or more package archives (which is true for a few built-in packages, but most importantly Org).

simply reading them from the file is straightforward enough. IMO if you want to be as robust and informative as possible, you should probably pursue that option, because otherwise you're depending on the package headers being correct and complete, and obviously that will not always be the case. :)

In my opinion, this is a very bad idea. I only made the proposal as a joke. There is no way to tell what the actual dependencies are by parsing the file, for several reasons:

• optional dependencies may still appear as require forms
• features may be required in a non-straightforward way
• there's no way in general to tell what package provides a given feature

Any steps down that road are definitively in the wrong direction given that there is a specific metadata format designed to provide exactly this information, for exactly the reason that parsing it out of the file is hopeless.

Sure, but that won't fix the issue that @Northcode is reporting, because that wouldn't cause a newer, non-built-in version of Org to be downloaded. At least, it shouldn't, right?

Doing that will fix the problem because yes, it will cause a newer, non-built-in version of Org to be downloaded. That change would be the only thing required to make everything work seamlessly, and I'll happily provide a pull request to that effect.

(Assuming the problem is indeed that Org 8.x is loaded, then Org 9.x is loaded on top of it, which causes errors.)

That's the problem, but it wouldn't happen if straight.el knew that org were a dependency of outorg, regardless of which order packages were registered in.

a lot of packages are maintained by authors who rarely check bug reports

OK, but that's no excuse for not reporting the bugs.

So I think this issue is much more common than you realize.

I concede this point. You know more than I. But I don't believe it's an intractable problem, and am perfectly happy to provide pull requests to fix package metadata every once in a while.

[alphapapa]

Yes, of course. Sorry, I intended my comment to be humorous. Just wanted to make a point :)

Sorry. Poe's law and all... :)

I suppose specification is rather a strong word, but I was referring to the documentation of the package system in the Emacs Manual and Emacs Lisp Reference Manual. The question isn't addressed specifically, so is subject to interpretation, but in the absence of a distinction between built-in packages and non-built-in packages I would presume that they are to be treated equivalently.

You might be right, but I'm looking at it from the other perspective: lacking a clear specification in the docs, I fall back on what's commonly done in practice--which IME usually means not declaring dependencies on built-ins.

Frankly, I think insisting on this would be a bit of a burden, as well: a package author, having developed a package that definitely works on a certain version of Emacs, the version number of which is obvious, the author would now have to dig out the version number of every built-in library file as well.

Also, are we certain that these version numbers are always incremented when changes are made? It wouldn't surprise me if changes are often made to them between Emacs releases without changing the library file's version header. Of course, I haven't researched this, so who knows. :) Anyway, I'm guessing that depending on these library version headers may not be a reliable way to determine compatiblity. (Hopefully I'm wrong, of course.)

Would a newer version of cl-lib even be expected to work on older Emacsen?

Answering that question is the purpose of the (emacs "21") dependency included in the Package-Requires header in cl-lib.el from GNU ELPA.

I'm a bit confused here, as this implies that cl-lib is intended to be independently upgradable between Emacs releases, but your next comment seems to indicate the opposite...

A newer version of cl-lib would likely use newer Emacs features, and since it's a built-in, I doubt it would be required to be backwards-compatible. e.g. loading cl-lib from Emacs 25.3 on Emacs 24.3 would probably not work.

That's true, but your comment only applies to the version shipped with Emacs, rather than the version provided on GNU ELPA. My comments concern when another version of a built-in package is also available from one or more package archives (which is true for a few built-in packages, but most importantly Org).

Do you mean that the version of cl-lib on ELPA is developed separately from the version included with Emacs? If not (and I doubt it is), then the (emacs "21") dependency header wouldn't seem to be meaningful or accurate. I mean, if I boot up a VM and install Emacs 21, will the current ELPA version of cl-lib work on it? That would be an impressive feat of backward compatibility! :) But IIUC the devs don't maintain compatibility for built-in packages that way.

Org is unique in that it is maintained and released separately from Emacs but also merged into Emacs's repo periodically and released with it. This does present some unique issues now and then.

In my opinion, this is a very bad idea. I only made the proposal as a joke. There is no way to tell what the actual dependencies are by parsing the file, for several reasons:

All of those reasons are true, of course. There's only so much we can do with the Emacs package system.

Sure, but that won't fix the issue that @Northcode is reporting, because that wouldn't cause a newer, non-built-in version of Org to be downloaded. At least, it shouldn't, right?

Doing that will fix the problem because yes, it will cause a newer, non-built-in version of Org to be downloaded. That change would be the only thing required to make everything work seamlessly, and I'll happily provide a pull request to that effect.

(Assuming the problem is indeed that Org 8.x is loaded, then Org 9.x is loaded on top of it, which causes errors.)

That's the problem, but it wouldn't happen if straight.el knew that org were a dependency of outorg, regardless of which order packages were registered in.

Ok, now this is the crux of the issue. Let's make sure we're on the same page here:

1. The problem is that Org 8.x is loaded, then Org 9.x is loaded on top of it, which causes errors.

2. Outorg/Outshine do not declare a dependency on Org in the package headers, so straight.el doesn't do anything with Org when it loads Outorg/Outshine.

3. Therefore if Outorg/Outshine are loaded before Org, the built-in version is loaded.

4. Then when straight.el loads org itself, it loads the latest version, ignoring any built-in version.

5. This seems to mean that if a version-number-less (org) dependency is declared in a package header, straight.el will get the latest version of Org, ignoring the built-in version.

Now I suppose this is a matter of opinion, but as a user, if I load a package that doesn't declare a dependency on a specific version of a package, I wouldn't expect my package manager to automatically download the latest version when there's already a version installed that satisfies the declared dependency. Indeed, I would not want that behavior, as I do not want packages to be automatically upgraded, because that inevitably causes breakage in my config which I then have to fix before I can use Emacs again. Now I can see why some users would prefer that, and why that might be your preferred mode of operation for straight.el, but it's definitely an incompatibility with package.el, the existing Emacs package manager.

Therefore this still seems to boil down to straight.el not maintaining backward compatibility with package.el, and pushing that burden onto package maintainers.

a lot of packages are maintained by authors who rarely check bug reports

OK, but that's no excuse for not reporting the bugs.

1. But lack of declared dependencies on built-in libraries is not necessarily a bug. In fact, I think few package authors would think so. Consider that, even though MELPA reviews package contents when new recipes are submitted, and Steve gives feedback as detailed as asking authors to use when and unless instead of single-conditional ifs, I've never seen Steve or anyone else tell a package author to declare a dependency on built-in packages.

2. You can report the issues, but IMO there are quite a few packages that will simply not be updated to fix them. There are issues and PRs I've sent on some MELPA packages many months ago that have still received no reply. It's just the way things are.

So I think this issue is much more common than you realize.

I concede this point. You know more than I. But I don't believe it's an intractable problem, and am perfectly happy to provide pull requests to fix package metadata every once in a while.

I think you are much more knowledgable about how the Emacs package systems work than I am! :) Maybe I have looked at more MELPA packages, who knows.

In this case, AFAIK if I add a version-less dependency on Org, it won't do any harm to existing users. But at the same time, I'm reluctant to make changes to workaround incompatible behavior in alternative package managers, especially when it can be fixed by the user by simply adjusting his init file. I've had to do the same thing for my init file, which still uses package.el. In some cases, package loading order does matter.

So are we on the same page? Let me know if I've misunderstood anything. Thanks.

[raxod502]

Frankly, I think insisting on this would be a bit of a burden, as well: a package author, having developed a package that definitely works on a certain version of Emacs, the version number of which is obvious, the author would now have to dig out the version number of every built-in library file as well.

Well, they don't have to figure out what version number is needed, since package.el lets you declare a dependency on a certain library without specifying the version number.

Do you mean that the version of cl-lib on ELPA is developed separately from the version included with Emacs?

I think my example of cl-lib was a poor choice. The actual purpose of cl-lib on GNU ELPA is to backport cl-lib (which is only available in Emacs 24.3+) to older Emacs versions.

There may not currently be any package archives that exercise the ability of package.el to include both built-in and available versions of the same package in its dependency resolution algorithm. It may indeed just be a hypothetical. I was just saying that it is an intended feature of package.el.

Org is unique in that it is maintained and released separately from Emacs but also merged into Emacs's repo periodically and released with it. This does present some unique issues now and then.

Right. And I have seen so many threads of people struggling to get Org installed correctly with package.el. It's much easier with straight.el (at least if dependencies are declared).

Ok, now this is the crux of the issue. Let's make sure we're on the same page here

Your summary is correct.

Now I suppose this is a matter of opinion, but as a user, if I load a package that doesn't declare a dependency on a specific version of a package, I wouldn't expect my package manager to automatically download the latest version when there's already a version installed that satisfies the declared dependency. Indeed, I would not want that behavior, as I do not want packages to be automatically upgraded, because that inevitably causes breakage in my config which I then have to fix before I can use Emacs again.

Here are a couple of relevant points which may change your perspective on this behavior:

• straight.el does not consider built-in packages as actually installed, except as a last resort if they cannot be found elsewhere. The main reason for this is that using built-in packages is an inferior user experience: they cannot be easily edited, version-controlled, automatically recompiled, or contributed upstream. Therefore in the view of straight.el, no "upgrade" is happening: Org is being installed for the first time.
• straight.el does not upgrade packages automatically. Very, very, very far from it. One of the specific aims of straight.el is that your configuration never ever ever breaks due to package changes. And it actually does a far better job of achieving this than package.el. With package.el, there's no way to install an older version of a package (i.e. the same one that you had installed on a different machine). With straight.el, once you save a version lockfile, then the revisions specified there are automatically checked out on every new machine, and you are using exactly the same Git revisions of everything. There's no way your configuration can break due to upstream changes without something traumatic like an entire Git repository disappearing or being rebased. (And in those cases, you'll be informed.)
• Putting this together, what happens with Org is that straight.el downloads an initial version of it so as to get it into the straight.el build system (which, as I said, has a superior user experience). And then that's it—no upgrades happen automatically, and if you save a version lockfile, then even when you move to a separate machine, you'll be using the same revision.

(Another interesting point is that straight.el actually completely ignores version numbers! I've never had anyone complain about this, so I think my judgement that they don't actually provide enough utility to justify complicating dependency resolution was a good one. Perhaps that will also change your view of installing a new version of Org as an "upgrade"?)

Therefore this still seems to boil down to straight.el not maintaining backward compatibility with package.el

Well... yes and no. The fact is, getting Org to work with package.el is a huge pain. straight.el eliminates this problem by using a different package management model, which is not entirely backwards compatible with package.el. So you either have to have Org be a huge pain, or have it work well but rely more heavily on package metadata.

Of course, the real solution is to not have packages be built-in, and instead have a better package manager so that using packages is as easy as using builtins. But it's taken me six months to convince emacs-devel not to have package.el edit the init-file at startup to insert (package-initialize)—I don't even want to think about proposing a change like this :P

But lack of declared dependencies on built-in libraries is not necessarily a bug. In fact, I think few package authors would think so. Consider that, even though MELPA reviews package contents when new recipes are submitted, and Steve gives feedback as detailed as asking authors to use when and unless instead of single-conditional ifs, I've never seen Steve or anyone else tell a package author to declare a dependency on built-in packages.

OK, fair enough. I personally would place missing dependency information in the same category as other missing metadata (like Commentary), in that it's not exactly a bug, just something that could be annoying. I think the reason that nobody comments on it is because before straight.el there weren't any tools that relied upon this metadata.

You can report the issues, but IMO there are quite a few packages that will simply not be updated to fix them.

OK. If a package doesn't get ever updated, I would consider it dead and not worthy of too much concern. Like I said earlier, there's really no way to generate this metadata if it's not provided.

if I add a version-less dependency on Org, it won't do any harm to existing users

Yes, I can't think of a case where this could cause any harm.

I'm reluctant to make changes to workaround incompatible behavior in alternative package managers

I object to your use of the word workaround. It implies that straight.el is doing something wrong. It's not that straight.el is refusing to accept the package.el format, it's actually that straight.el requires a stricter compliance to the package.el format than even package.el requires. The word workaround also implies that the change is specifically necessary for straight.el. But it is also useful for any other tool that uses the metadata in Package-Requires.

Arguably, even though adding a dependency on org is not necessary, it would still be more correct.

especially when it can be fixed by the user by simply adjusting his init file

Sort of. But, consider: I might want to try outorg temporarily by running M-x straight-use-package RET outorg RET. And then I would cry, because if I had not previously loaded Org then things would be broken pretty much beyond repair in my current session.

And as far as what it looks like in the init-file, suppose that I don't care about Org itself. I only care about outorg. Then I would just have:

(straight-use-package 'outorg)

And this wouldn't work, because of the missing dependency information. I'd have to adjust it to:

(straight-use-package 'org)
(straight-use-package 'outorg)

I basically have to rewrite the dependency information in my init-file. And whenever the user is required to do additional work to use a package, it really implies to me that perhaps the package should be doing this work instead.

So are we on the same page?

Partially. I do agree that missing built-in dependencies is probably more prevalent and not as big of a problem than I made it out to be. But I still think it's worthwhile to fix the issue in popular packages, because it makes the lives of some users more convenient while not causing any problems for the rest.

Thanks for your patience. It's very good to discuss these things. Reminds me of the great Friendly discussion about (package-initialize), which is possibly getting close to finished after half a year... ❤️

[alphapapa]

Well, they don't have to figure out what version number is needed, since package.el lets you declare a dependency on a certain library without specifying the version number.

Right, but straight.el handles that case differently: If a built-in version of that dependency already exists, package.el lets it satisfy the unversioned requirement, while straight.el installs the latest version. Of course, we hope that the latest version doesn't break backward compatibility, but since there is no enforcement or even consensus among package authors, this is not guaranteed.

Maybe what we're seeing here is sort of a paradigm shift: from versioned, packaged repositories to direct-from-git virtual repositories, and perhaps some conflicts in behavior are inevitable (at least, without a lot more coding to work around them).

The actual purpose of cl-lib on GNU ELPA is to backport cl-lib (which is only available in Emacs 24.3+) to older Emacs versions.

That makes sense.

There may not currently be any package archives that exercise the ability of package.el to include both built-in and available versions of the same package in its dependency resolution algorithm. It may indeed just be a hypothetical. I was just saying that it is an intended feature of package.el.

Well, I think MELPA does. For example, IIRC some of my Org packages require Org 9, and they declare a dependency on it, so if the user's built-in version is Org 8.x, package.el should install Org 9 if it's available in a configured repo. It's not very widely used outside of Org though, AFAIK, because typically if a package requires a certain version of a built-in, it will probably declare a dependency on the version of Emacs that includes it rather than the package itself.

Right. And I have seen so many threads of people struggling to get Org installed correctly with package.el. It's much easier with straight.el (at least if dependencies are declared).

That's good to know. It would also help if the org-plus-contrib package were folded into plain org; maybe someday...

straight.el does not consider built-in packages as actually installed, except as a last resort if they cannot be found elsewhere.

That sounds good.

straight.el does not upgrade packages automatically.

Glad to know that.

With straight.el, once you save a version lockfile, then the revisions specified there are automatically checked out on every new machine, and you are using exactly the same Git revisions of everything.

Is there an option to make the lockfile automatically?

There's no way your configuration can break due to upstream changes without something traumatic like an entire Git repository disappearing or being rebased. (And in those cases, you'll be informed.)

It's because of those scenarios that I keep ~/.emacs.d, including ~/.emacs.d/elpa in git. :) Does straight.el support that?

Putting this together, what happens with Org is that straight.el downloads an initial version of it so as to get it into the straight.el build system (which, as I said, has a superior user experience). And then that's it—no upgrades happen automatically, and if you save a version lockfile, then even when you move to a separate machine, you'll be using the same revision.

Well, that sounds pretty good, but I'm not sure that's the best way to do it. For a lot of packages that's probably fine, but Org does make backward-incompatible changes somewhat regularly. For example, Org 9 made backward-incompatible changes not only to behavior but to the Org file format; users who didn't read the release notes and run the conversion tool would suffer loss of functionality as e.g. planning lines would become invisible to Org. There are also smaller incompatible changes made more frequently in minor releases. So if I were a user on a version of Org, and then started using straight.el, I would definitely not want Org to be upgraded to the current release automatically.

(Another interesting point is that straight.el actually completely ignores version numbers! I've never had anyone complain about this, so I think my judgement that they don't actually provide enough utility to justify complicating dependency resolution was a good one.

Well, that's interesting, and the, shall we say, lassiez-faire style that the Emacs package community tends to develop a lot of non-core or non-library packages gets away with that a lot of the time. But there are plenty of counterexamples, such as Org, Magit (which once required a certain version of git upon upgrade, which would then break magit until you manually installed a new version of git), dash.el (which plans to make backward-incompatible changes soon), etc. I'm glad that it works a lot of the time, but I think trouble is inevitable.

Perhaps that will also change your view of installing a new version of Org as an "upgrade"?)

Well, as I explained, Org is a particularly strong example of a package that should only be upgraded carefully. :)

The fact is, getting Org to work with package.el is a huge pain. straight.el eliminates this problem by using a different package management model, which is not entirely backwards compatible with package.el. So you either have to have Org be a huge pain, or have it work well but rely more heavily on package metadata.

Well, this might be too nit-picky to bother talking about, but I'm not sure I understand or agree exactly here. The only problem I've ever had with Org and package.el is caused by there being both org and org-plus-contrib packages, and that problem isn't related to the problem we've been discussing.

And as you explained, straight.el is not accounting for version numbers, so I'm not sure it's necessarily a fair comparison between it and package.el. IMO straight.el ignores version numbers at its peril. ;) Luckily, it works most of the time, but it will inevitably fail in some cases, because the version numbers are there for a reason.

Of course, the real solution is to not have packages be built-in, and instead have a better package manager so that using packages is as easy as using builtins.

Well, you probably don't want to go down that rabbit hole here, but I'm not sure what you mean about not having packages be built-in. I don't think it would be an improvement to distribute built-ins separately in most cases. There might be a few that would be good to split out, but stuff like cl-lib, seq, map? That doesn't seem like a good idea to me.

But it's taken me six months to convince emacs-devel not to have package.el edit the init-file at startup to insert (package-initialize)—I don't even want to think about proposing a change like this :P

Yeah, I saw that thread yesterday but I didn't read all of it. I guess the point is to not automatically use package.el, so that alternative package managers can be used?

OK, fair enough. I personally would place missing dependency information in the same category as other missing metadata (like Commentary), in that it's not exactly a bug, just something that could be annoying. I think the reason that nobody comments on it is because before straight.el there weren't any tools that relied upon this metadata.

Not sure exactly what you mean here. package.el uses it, MELPA uses it, Borg/epkg use it, etc.

OK. If a package doesn't get ever updated, I would consider it dead and not worthy of too much concern.

Well, they are concerns for users who use them. :) As the EmacsWiki/MELPA discussion has shown, there are a lot of rarely-updated packages that are widely used. Emacs packages also have more of a tendency to get to a virtually-done state than other systems, to the point where they rarely need updates (something I appreciate!).

Like I said earlier, there's really no way to generate this metadata if it's not provided.

Yeah, it would be quite an undertaking to do some kind of recursive static analysis across packages. Theoretically possible I guess, but not something I want to work on! :)

I object to your use of the word workaround. It implies that straight.el is doing something wrong. It's not that straight.el is refusing to accept the package.el format, it's actually that straight.el requires a stricter compliance to the package.el format than even package.el requires.

I think I disagree slightly here. Either the specification requires something or it doesn't; either it permits something or it doesn't. If straight.el adds additional requirements, I think that's a case of it being incompatible with the specification. As an analogy, consider HTML parsers (an enormous world of pain). Browsers go to extreme length to handle poorly-formed documents. Obviously I'm not advocating the equivalent here, but if a browser were to add an additional requirement on top of the HTML spec, that wouldn't be a case of the browser requiring stricter compliance with the spec, but adding to and changing the spec in an incompatible way. The result would be that people would stop using that browser, not that web sites would all make changes to accomodate it. (Of course, you could cite IE and its IE-specific features as a counter-example, but I'm sure you see my point.)

The word workaround also implies that the change is specifically necessary for straight.el. But it is also useful for any other tool that uses the metadata in Package-Requires.

Well, in this case, it is specifically necessary for straight.el, because nothing else is needing the change to be made. :) AFAIK the general expectation is that packages need not declare dependencies on built-ins unless they differ from the version included with the version of Emacs they are intended to be used with (in which case it's arguable which version of Emacs they are intended to be used with, but I digress).

So this still seems to me like a case of straight.el asking packages to change to suit its backward-incompatible behavior.

Arguably, even though adding a dependency on org is not necessary, it would still be more correct.

If you think that all packages should declare dependencies on all built-in libraries, sure. If not, then no. :)

Sort of. But, consider: I might want to try outorg temporarily by running M-x straight-use-package RET outorg RET. And then I would cry, because if I had not previously loaded Org then things would be broken pretty much beyond repair in my current session.

I don't understand why that would be the case. In that case, Outorg would (require 'org), which would load the built-in version of Org, which would work fine with Outorg. A problem would only arise if you later ran M-x straight-use-package RET org RET and straight.el then loaded the latest Org on top of the version already loaded in the Emacs image. Maybe it should do (featurep 'org) before trying to load on top of an already-loaded package. That seems like a bug in straight.el to me.\

And as far as what it looks like in the init-file, suppose that I don't care about Org itself. I only care about outorg. Then I would just have:

(straight-use-package 'outorg)

And this wouldn't work, because of the missing dependency information.

As far as I know, that would work fine.

I basically have to rewrite the dependency information in my init-file. And whenever the user is required to do additional work to use a package, it really implies to me that perhaps the package should be doing this work instead.

Ideally the web of dependencies would be accurate and comprehensive, and users could attempt to load packages in any order, and they would always be resolved correctly. Hopefully that will be achieved someday in the Emacs package ecosystem. And maybe straight.el or similar projects will lead the way.

Partially. I do agree that missing built-in dependencies is probably more prevalent and not as big of a problem than I made it out to be. But I still think it's worthwhile to fix the issue in popular packages, because it makes the lives of some users more convenient while not causing any problems for the rest.

That seems reasonable. I don't mind adding a versionless dependency to the headers. Although I confess, with all the packages I maintain on MELPA, if you start asking me to do this for all of them, I may become annoyed. ;) That is to say, if the community were to settle on straight.el or some other package manager as the standard one, I would gladly adjust my packages to comply with its requirements; but I'm not eager to support a multitude of them, each with their own quirks. I think one of the strengths of Emacs and its packages is that they are relatively stable and backward-compatible, and so I think new package managers should strive to be as compatible as possible with the standard package manager, even when that means compatibility with irritating quirks in package.el.

Thanks for your patience. It's very good to discuss these things. Reminds me of the great Friendly discussion about (package-initialize), which is possibly getting close to finished after half a year...

Yes, and thanks to you as well. I hope that projects like yours will lead the way to improving the Emacs package system. It takes enthusiasm and dedication to see a project like straight.el through. I hope that in the future the community can either bring package.el up to speed or standardize on another package manager. Until then, issues like this will persist.

[raxod502]

Is there an option to make the lockfile automatically?

I hadn't considered that because it seems like it would be annoying. You couldn't have local modifications in your repositories, because saving a lockfile means first getting all of your local repositories into states that can be replicated on other machines (for reproducibility).

It's because of those scenarios that I keep ~/.emacs.d, including ~/.emacs.d/elpa in git. :) Does straight.el support that?

Yes.

It would also help if the org-plus-contrib package were folded into plain org; maybe someday...

I thought the main reason the two were separate was that it takes much longer to byte-compile all of the contrib files, and not everyone wants to do that. So it seems doubtful to me that the two will (or should) merge.

So if I were a user on a version of Org, and then started using straight.el, I would definitely not want Org to be upgraded to the current release automatically.

But if you were switching to straight.el and you wanted to avoid incompatibilities, then right after the initial installation you would just check out whatever revision corresponded to the one you were using previously. This is only ever a problem on initial setup.

Consider also the fact that package.el doesn't handle this case at all. If you are a user on a version of Org, and then copied your Emacs configuration to another machine, package.el would install the latest version! The only way you avoid this is keeping ~/.emacs.d/elpa in version control. Whereas if you're using straight.el, the same revision will be used on all machines, without having to keep any packages in version control.

But there are plenty of counterexamples, such as Org, Magit (which once required a certain version of git upon upgrade, which would then break magit until you manually installed a new version of git), dash.el (which plans to make backward-incompatible changes soon), etc. I'm glad that it works a lot of the time, but I think trouble is inevitable.

But the trouble is not really trouble. One must ask, when does the trouble arise? And the answer is, when you upgrade your packages. Because with straight.el, there is no other time that the revision of a package is changed, other than when you explicitly ask to upgrade. And of course when you upgrade a package you'll expect things to break if you don't read the release notes. In every other circumstance, everything is frozen and there's no way things can break.

Well, this might be too nit-picky to bother talking about, but I'm not sure I understand or agree exactly here. The only problem I've ever had with Org and package.el is caused by there being both org and org-plus-contrib packages, and that problem isn't related to the problem we've been discussing.

The problems I was thinking of were like https://github.com/jwiegley/use-package/issues/319.

IMO straight.el ignores version numbers at its peril. ;)

But to reiterate, the only time when version numbers will make a difference for package.el is during initial installation of a package. And Emacs packages, for better or for worse, usually require some manual work to set up the first time. Part of this work with straight.el would presumably be making sure that the version that was installed works well enough, and otherwise checking out a different revision.

I don't think it would be an improvement to distribute built-ins separately in most cases. There might be a few that would be good to split out, but stuff like cl-lib, seq, map? That doesn't seem like a good idea to me.

I agree that cl-lib, seq, map, etc. should be part of the standard library of Emacs. But if you actually look at Lisp code shipped with Emacs, the vast majority of it doesn't bear any resemblance to a standard library. Looking just at the top of the list:

  allout             2.3           built-in   extensive outline mode for use alone and with other modes
  allout-widgets     1.0           built-in   Visually highlight allout outline structure.
  ansi-color         3.4.2         built-in   translate ANSI escape sequences into faces
  antlr-mode         2.2.3         built-in   major mode for ANTLR grammar files
  artist             1.2.6         built-in   draw ascii graphics with your mouse
  auth-source-pass   2.0.0         built-in   Integrate auth-source with password-store
  bs                 1.17          built-in   menu for selecting and displaying buffers
  cedet              2.0           built-in   Setup CEDET environment
  cfengine           1.4           built-in   mode for editing Cfengine files
  chart              0.2           built-in   Draw charts (bar charts, etc)

Not a single one of these could conceivably be considered as part of the category of standard-library packages like cl-lib, seq, map. Why is it all bundled? Well, when these packages were put into Emacs, we didn't have a package manager!!

I found it interesting that XEmacs did this, once upon a time.

Yeah, I saw that thread yesterday but I didn't read all of it. I guess the point is to not automatically use package.el, so that alternative package managers can be used?

No, package.el is still used automatically. The point of the change was to prevent Emacs from automatically editing the init file at startup, which was ridiculously gauche. This does have the positive side effect that the way to use an alternative package manager is less dumb (put (setq package-enable-at-startup nil) in the early init file, rather than putting a commented-out ;; (package-initialize) in the regular init file).

Not sure exactly what you mean here. package.el uses it, MELPA uses it, Borg/epkg use it, etc.

They don't use the dependencies on built-ins, though. That's the part that's under discussion here.

As the EmacsWiki/MELPA discussion has shown, there are a lot of rarely-updated packages that are widely used.

Well, those packages are doomed to die a slow death as functions are deprecated and packaging requirements are modified. Fine by me; if people really care then somebody will become the new maintainer.

Yeah, it would be quite an undertaking to do some kind of recursive static analysis across packages. Theoretically possible I guess, but not something I want to work on! :)

Actually, very much not theoretically possible. Determining whether Elisp code is going to require a feature is equivalent to the halting problem. And then there's the issue that it's impossible in general to correctly guess what package provides any given feature. And even this secondary problem is not solvable even in theory, since multiple different packages could provide different versions of the same feature (e.g. org and org-plus-contrib).

Browsers go to extreme length to handle poorly-formed documents.

This point is reasonable. I'd however argue that our situation here is better because the average Emacs user is more resourceful than the average web browser user, and the average Emacs package maintainer is more resourceful than the average web site maintainer. So we can expect to get things working for everyone without quite so many hacks.

A problem would only arise if you later ran M-x straight-use-package RET org RET and straight.el then loaded the latest Org on top of the version already loaded in the Emacs image.

Well... I should clarify what I mean by "broken". From the perspective of a straight.el user, the only correct way to load Org is via straight.el. Therefore if the built-in version from Emacs gets loaded at any point, then there's an issue.

Maybe it should do (featurep 'org) before trying to load on top of an already-loaded package. That seems like a bug in straight.el to me.

It should be clarified that at no point is straight.el explicitly loading Org. Org gets loaded as a consequence of byte-compiling outorg; there's no way to avoid that.

So I assume what you mean is that if the org feature is already loaded, then straight.el should warn when making the org package available. Unfortunately, there's no general way to tell what features a package provides. Although it is likely that heuristics could be used with great success here, I'd rather not.

But your suggestion gave me an idea about how to change the result of missing built-in dependencies from subtle undefined behavior to an explicit error: https://github.com/raxod502/straight.el/issues/236.

I don't mind adding a versionless dependency to the headers. Although I confess, with all the packages I maintain on MELPA, if you start asking me to do this for all of them, I may become annoyed. ;)

Fair enough. Given that this change benefits me primarily, I'd be happy to provide the pull requests for other packages that people may find lacking dependency information.

and so I think new package managers should strive to be as compatible as possible with the standard package manager, even when that means compatibility with irritating quirks in package.el.

OK, I think this is just an irreducible difference in philosophy, since I believe the opposite. But that's alright, we can agree to disagree.

[alphapapa]

I hadn't considered that because it seems like it would be annoying. You couldn't have local modifications in your repositories, because saving a lockfile means first getting all of your local repositories into states that can be replicated on other machines (for reproducibility).

So after a lockfile is set up, the user can't make local changes?

I thought the main reason the two were separate was that it takes much longer to byte-compile all of the contrib files, and not everyone wants to do that. So it seems doubtful to me that the two will (or should) merge.

For local building, a Makefile target can be used to select which set of files to build. For installations from repos, the contrib files should be put into a separate package. Having two packages which install the same set of files has caused so many man-hours of wasted problem solving...

But if you were switching to straight.el and you wanted to avoid incompatibilities, then right after the initial installation you would just check out whatever revision corresponded to the one you were using previously. This is only ever a problem on initial setup.

Okay, so that's something that straight.el users will just have to deal with manually, I guess.

Consider also the fact that package.el doesn't handle this case at all. If you are a user on a version of Org, and then copied your Emacs configuration to another machine, package.el would install the latest version! The only way you avoid this is keeping ~/.emacs.d/elpa in version control.

Yes, that's why I always advocate doing that. I think doing otherwise is madness. :) Otherwise, with a substantial number of packages, every time you install your Emacs config on a new system, or every time you upgrade the set of packages on one system, you're going to have a unique set of incompatibilities to solve before you can use Emacs again.

Whereas if you're using straight.el, the same revision will be used on all machines, without having to keep any packages in version control.

Well, actually all of those packages are in version control, they're just in repos out of the user's control. :)

Because with straight.el, there is no other time that the revision of a package is changed, other than when you explicitly ask to upgrade.

IIUC they can also change when setting up on a new system--unless the user has set up a lockfile, but IIUC that precludes customization, so... I confess that I'm getting a bit confused now. The simplicity of using package.el with /elpa in version control appeals to me here. ;) But you don't need to sell me on straight.el here.

But to reiterate, the only time when version numbers will make a difference for package.el is during initial installation of a package.

Are you sure? What about this scenario:

1. User installs A, which requires Z, which is then installed as a dependency.
2. User later installs A+1, which requires Z+1.
3. Does package.el not then upgrade Z?

Why is it all bundled? Well, when these packages were put into Emacs, we didn't have a package manager!!

Sure, so maybe some of them will gradually be moved to ELPA or a separate package (of course some distros already split some of them into separate binary packages).

No, package.el is still used automatically. The point of the change was to prevent Emacs from automatically editing the init file at startup,

Without adding (package-initalize) to the init file, users won't be able to load installed packages, right?

They don't use the dependencies on built-ins, though. That's the part that's under discussion here.

Are you sure? For example, AFAIK package.el will allow the built-in version of Org to satisfy a declared dependency in the headers.

Well, those packages are doomed to die a slow death as functions are deprecated and packaging requirements are modified. Fine by me; if people really care then somebody will become the new maintainer.

It's not fine by their users, though. And, surprisingly, a lot of Emacs users are not elisp programmers, package developers, etc. There are a lot of Emacs users who don't customize their init files, or who do so only by pasting snippets they find online. We shouldn't pull the rug out from under these users. I'm talking about packages which have stabilized to the point where they rarely, if ever, need updates. You seem to be proposing to invalidate those packages because of package headers, not any actual breakage in the package's code. That's not okay.

Actually, very much not theoretically possible. Determining whether Elisp code is going to require a feature is equivalent to the halting problem.

To do so perfectly, sure. But there are only so many functions that load code in Emacs. It's not necessary to execute a package's code to determine what it may eventually load.

And then there's the issue that it's impossible in general to correctly guess what package provides any given feature

It's true that some packages contain files that don't have the same name as the package itself, but that doesn't mean that some basic checks would be useless. Nor does it mean that they would be useful enough to spend time on, haha. But I would again point to package-lint as something to consider in this respect.

I'd however argue that our situation here is better because the average Emacs user is more resourceful than the average web browser user, and the average Emacs package maintainer is more resourceful than the average web site maintainer. So we can expect to get things working for everyone without quite so many hacks.

We're part of a larger community and ecosystem. As the more practiced participants, we have a duty as stewards to be good caretakers for those participants who are simply users of it. That means we should avoid breaking things without a very good reason (e.g. security, safety, or really old code that virtually no one uses and would be a significant burden to maintain compatibility with).

By the way, some of these non-elisp-programmer, "simple users" are actually people like scientists, statisticians, software developers in other languages, writers, etc. Just because they don't have expertise in the specific domain of Emacs or Elisp or Emacs packages doesn't mean that they are unintelligent or incapable--it means that they have other domain expertise and are busy using it. If we break the code that they are using and have been using for years, and force them to devote time to fixing problems in Emacs or finding alternative solutions, we are doing them a diservice, and we are harming the Emacs community in the long-term.

For myself, one of the reasons I use Emacs is that, even as surrounding software in the FOSS community rapidly changes and breaks incessantly, Emacs doesn't. We should be careful not to mess with this. So this is why, even as I applaud your work on improving package management, I caution you to do so in a way that provides long-term stability and compatibility, and considers the needs of all users. This is necessary if you want your alternative to be widely useful rather than for a niche set of users (Emacs users already being a niche set of computer users).

Well... I should clarify what I mean by "broken". From the perspective of a straight.el user, the only correct way to load Org is via straight.el. Therefore if the built-in version from Emacs gets loaded at any point, then there's an issue.

This is a bit confusing to me. Does straight.el not support loading built-in packages? What about ones which are only available as built-ins?

It should be clarified that at no point is straight.el explicitly loading Org.

Sorry, I'm confused again: you mean if the user does M-x straight-use-package RET org RET, straight.el doesn't load Org? Or by "at no point" do you mean something else?

So I assume what you mean is that if the org feature is already loaded, then straight.el should warn when making the org package available. Unfortunately, there's no general way to tell what features a package provides. Although it is likely that heuristics could be used with great success here, I'd rather not.

Sorry, I don't understand. What I'm saying is, if straight.el comes across a package that depends on org or does (require 'org), straight.el could check (featurep 'org) before actually loading it, to avoid problems like this.

OK, I think this is just an irreducible difference in philosophy, since I believe the opposite. But that's alright, we can agree to disagree.

Sure, if you have different goals for straight.el, that's up to you. I'm not sure exactly what they are, but if you are aiming to eventually replace package.el, I think it may be necessary to ensure closer compatibility with it. But I'm sure emacs-devel will take care of ensuring that. ;)

[raxod502]

So after a lockfile is set up, the user can't make local changes?

Not quite. When writing a lockfile, all local changes must be reconciled (by the user, interactively, with the aid of straight.el-generated popups) by committing and pushing them, or updating package recipes. This is to preserve the invariant that if you write a lockfile, then your configuration will be immediately portable to any machine, no strings attached.

The correct mental model is that your configuration's behavior is specified by (init file + lockfile + arbitrary local changes). And on a new machine, the behavior will be (init file + lockfile). So, you can only be sure that behavior won't change on a new machine if you first reconcile all the local changes. Since I want to discourage people building up large quantities of local changes on one machine (since it introduces the possibility for breakage), I have straight.el ask them to reconcile said changes when they write a lockfile.

For installations from repos, the contrib files should be put into a separate package.

This is obvious in hindsight and I wholeheartedly endorse your suggestion. Just have package org-contrib which declares a dependency on package org.

Okay, so that's something that straight.el users will just have to deal with manually, I guess.

Manual work here happens whenever you change from one package manager to another package manager, unless both package managers store their metadata in the same format on disk.

(It might be possible/interesting to create a tool in straight.el-support that takes a ~/.emacs.d/elpa directory and tries to produce a ~/.emacs.d/init.el and ~/.emacs.d/straight that would replicate the configuration, though.)

Yes, that's why I always advocate doing that. I think doing otherwise is madness. :)

One main reason I created straight.el was I wanted to avoid madness, but I also wanted to avoid checking ~/.emacs.d/elpa into version control.

IIUC they can also change when setting up on a new system--unless the user has set up a lockfile, but IIUC that precludes customization, so... I confess that I'm getting a bit confused now.

Right, lockfile and customization are not mutually exclusive except that straight.el actively discourages you from letting customizations persist for a long time. In my view, editing the source code of a package is a good debugging tool and should be allowed/well-supported, unlike in package.el, but in the long run you should fork the package and update the recipe.

The simplicity of using package.el with /elpa in version control appeals to me here. ;)

But then consider what this "simplicity" buys you when you want to contribute a patch upstream to one of your packages.

But you don't need to sell me on straight.el here.

But I want to! ;)

Are you sure? What about this scenario:

You might be right. I don't know what happens in that situation. In any case, existing packages aren't upgraded by straight.el when you install a new one. You may observe that this has the potential for breakage, if you need to upgrade your other packages to accommodate the dependencies of the new one. And to that I would say—yes, but simplicity trumps perfection, especially since I've never ever not even once in more than a year of active Emacs development seen a problem caused by this.

Sure, so maybe some of them will gradually be moved to ELPA or a separate package

Yep, that has happened already to some of the obsolete packages, and I think it's a great move.

Without adding (package-initalize) to the init file, users won't be able to load installed packages, right?

Wrong, it is no longer necessary to have (package-initialize) in the init file—that was the whole point of the patch. package-initialize is now called before loading the init file.

Are you sure? For example, AFAIK package.el will allow the built-in version of Org to satisfy a declared dependency in the headers.

You are right; what I meant was that no current software relies on dependencies on built-ins. That is, if they're missing, then the behavior of current software doesn't change noticeably, even though the information may be technically used.

We're part of a larger community and ecosystem. As the more practiced participants, we have a duty as stewards to be good caretakers for those participants who are simply users of it.

OK, this is a very fair and reasonable argument. I do agree that we want things to work out of the box as often as possible, for the benefit of these users.

But I still refuse to use heuristics to guess dependencies. So how about the following solution, instead: Introduce a new, community-maintained recipe repository that provides custom recipes for the few packages which

• don't have enough packaging information to work with straight.el, but
• are completely unmaintained, and yet
• have a significant active user base

so that these packages can work out of the box with straight.el even without upstream fixes? (Of course, the preferable option would be for fixes to be adopted upstream, if possible.) I think this solution satisfies both of our desires, i.e. to not use horrifying hacks in package managers, but to still be supportive of an ecosystem larger than just developers.

I wouldn't expect such a recipe repository to be a big maintenance burden, because the intersection of the three categories above is pretty darn small, and the fact that the packages are unmaintained means, of course, that the recipes don't need to be updated! And this is all assuming, of course, that we can't find an existing source of metadata that works well enough (maybe el-get?).

Incidentally, this is rather similar to how I added support for Microsoft Windows to straight.el, which (among other things) doesn't support symlinks. The analogy, of course, being that I need to make straight.el work with pre-existing systems which are at cross purposes with my architecture. I could have changed the architecture, but a better solution was to retain the architecture, and just emulate symlinks on the software layer.

you mean if the user does M-x straight-use-package RET org RET, straight.el doesn't load Org? Or by "at no point" do you mean something else?

I think we mean different things by "load". When I say load, I mean evaluate a file of Lisp code. straight-use-package does not load anything except an autoloads file. Instead, it makes a package available, by adding it to the load-path. Incidentally, my recently merged package-initialize patch, on the advice of emacs-devel, changes the terminology in the Emacs manual from "load" to "make available" when referring to evaluating autoloads and adding to load-path.

So in this sense, straight.el does not load Org at any point. The complication is that straight.el does byte-compile packages when building them, of course, and such byte-compilation may cause the code to be loaded anyway, at least until we byte-compile in a child process (see https://github.com/raxod502/straight.el/issues/76).

Does straight.el not support loading built-in packages?

In light of what I've said above, I'm not sure in which sense you are saying "load". But I am guessing that, regardless of the sense in which you said "load", the answer is no, since straight.el doesn't recognize built-in packages in any sense except that if you ask straight.el to make a package available, and it can't be found in any recipe repository, then it will signal an error unless a package by that name is built-in. (I think the code is fairly readable.)

What I'm saying is, if straight.el comes across a package that depends on org or does (require 'org), straight.el could check (featurep 'org) before actually loading it, to avoid problems like this.

And the problem with this suggestion is that straight.el in fact never does (require 'org) at any point in loading either org or any package which depends on it, except incidentally in the process of byte-compiling a package (which is outside the control of straight.el except by advising require).

if you are aiming to eventually replace package.el, I think it may be necessary to ensure closer compatibility with it.

We shall see, indeed...

[alphapapa]

The correct mental model is that your configuration's behavior is specified by (init file + lockfile + arbitrary local changes). And on a new machine, the behavior will be (init file + lockfile). So, you can only be sure that behavior won't change on a new machine if you first reconcile all the local changes. Since I want to discourage people building up large quantities of local changes on one machine (since it introduces the possibility for breakage), I have straight.el ask them to reconcile said changes when they write a lockfile.

Thanks, that makes sense. It's actually very similar to the model I use, where my config's behavior is specified by init.el, custom.el, and elpa/. Most of the time I use a single machine, so sometimes I carry local changes without committing and/or pushing them for a while. Occasionally that can lead to minor annoyances, but that's my own fault, not the fault of the model. Rarely do I edit any package's code in elpa/ directly; as much as possible, I use Emacs's facilities to customize, like advice. If I do edit a package, I usually submit a PR and carry the changes locally until PR is merged. I like that you ask to reconcile changes regularly.

This is obvious in hindsight and I wholeheartedly endorse your suggestion. Just have package org-contrib which declares a dependency on package org.

Indeed. :) Hopefully someday...

Manual work here happens whenever you change from one package manager to another package manager, unless both package managers store their metadata in the same format on disk.

I see, thanks.

One main reason I created straight.el was I wanted to avoid madness, but I also wanted to avoid checking ~/.emacs.d/elpa into version control.

This is curious to me. If you don't mind me asking, why do you want to avoid that? I like that I have all my config's source code, including all packages, in my own repos, and that I don't depend on MELPA or ELPA or any GitHub repo being available to reproduce my config.

But then consider what this "simplicity" buys you when you want to contribute a patch upstream to one of your packages.

Well, I've heard that Borg is specifically designed to address this problem. Other than that, I do send PRs sometimes, and then when the package is updated, I just update it and overwrite any local changes I may have made.

But I want to! ;)

And you may. :) I am definitely interested, but honestly I don't feel like I have the energy to learn a new package manager and convert my existing config anytime soon. It will require some learning and adjustment, and what I have now works very well for me. But I'm glad you're working on improving things.

You might be right. I don't know what happens in that situation. In any case, existing packages aren't upgraded by straight.el when you install a new one. You may observe that this has the potential for breakage, if you need to upgrade your other packages to accommodate the dependencies of the new one. And to that I would say—yes, but simplicity trumps perfection, especially since I've never ever not even once in more than a year of active Emacs development seen a problem caused by this.

Well now that is interesting. I'm pretty sure that I would have experienced breakage like this, because it already happens to me with MELPA and package.el sometimes. But I tend to avoid upgrading packages until I need to, whereas it seems that many users always upgrade everything. Whether you upgrade as necessary or live on the bleeding edge, you have to deal with breakage of one kind or another.

Wrong, it is no longer necessary to have (package-initialize) in the init file—that was the whole point of the patch. package-initialize is now called before loading the init file.

Ah, I see. That sounds good.

But I still refuse to use heuristics to guess dependencies.

Yeah, I don't necessarily think it would be good or useful enough to bother with, either.

So how about the following solution, instead: Introduce a new, community-maintained recipe repository that provides custom recipes for...

That's an interesting idea. I guess whether it's worth it it would depend on how many users straight.el has and how many of them use those packages. I also wonder how existing efforts fit into the picture, like MELPA, emacsmirror, emacsattic, etc.

It sounds a little bit like Gentoo/Arch/Nix/Guix but for Emacs packages. But then I also wonder how the existing package recipe system fits in. I'm not expert enough on all that, but it's definitely an interesting idea.

straight-use-package does not load anything except an autoloads file. Instead, it makes a package available, by adding it to the load-path. ... The complication is that straight.el does byte-compile packages when building them, of course, and such byte-compilation may cause the code to be loaded anyway....And the problem with this suggestion is that straight.el in fact never does (require 'org) at any point in loading either org or any package which depends on it, except incidentally in the process of byte-compiling a package (which is outside the control of straight.el except by advising require).

Now I understand. Thanks.

[raxod502]

why do you want to avoid that?

I dislike checking other people's code into my repositories for the same reason that I dislike checking build products into my repositories. It solves some problems, but it's just so ugly :)

Plus, embedded packages are strictly inferior to individual repositories for development. (Check out old revisions, look at the logs, compute diffs and patches, send pull requests, merge changes...) In my experience, whenever someone who uses package.el decides to do serious development on a package, they clone the package separately and then make it available manually. I want to be able to jump into "serious developer mode" at any time. So, if developing a package requires cloning the repository and doing X, Y, Z—why not have the package manager just clone the repository and do X, Y, Z in the first place?

I don't depend on MELPA or ELPA or any GitHub repo being available to reproduce my config

Yes, that is a minor disadvantage of my way. I say minor because it's never caused any problems for me. The flexibility (including being able to implement fully lazy installation where packages are installed as autoloads are triggered, as well as use the full power of Git to move between package revisions, which I view as essential for development) outweighs the potential trouble, for me.

Well, I've heard that Borg is specifically designed to address this problem.

Yes, and so is straight.el. Except that Borg is not designed to be a full package manager ("Borg is a bare-bones package manager for Emacs packages. It provides only a few essential features and should be combined with other tools"), while straight.el is.

I guess whether it's worth it it would depend on how many users straight.el has and how many of them use those packages

I figure that lack of users and/or usage is a self-correcting problem. If people don't use the problematic packages, then those packages don't need to have recipes in the repository. Contrariwise, all it takes for a package to get fixed is for one person to say

1. I want to use package X!
2. It doesn't work. Why?
3. Ah, the default recipe from EmacsMirror didn't work. Let me just change Y...
4. It works now. Oh, might as well put it in the public repository for everyone else.

I also wonder how existing efforts fit into the picture, like MELPA, emacsmirror, emacsattic, etc.

The role of MELPA is reducing the number of recipes which need to be maintained from 100% to 20%. The role of EmacsMirror is reducing the number from 20% to 5%. The only packages that need custom recipes here is the few packages that are (1) not in MELPA and (2) too complicated for the default EmacsMirror packaging to work out of the box. I.e. the packages which don't already have a perfectly good recipe in some other recipe repository. Again I think of el-get which may reduce the number of problem cases to an even smaller number.

But then I also wonder how the existing package recipe system fits in.

It's actually very simple. The variable straight-recipe-repositories gives a list of recipe repository names. A recipe is provided for each repository, to clone it (melpa or epkgs). Then when a package name is given, each recipe repository is checked in turn to see if it provides a recipe for that package. The first one found is used; if none are found, you must give your own. You can also override a found recipe by giving your own.

Just to clarify, this would require absolutely no additional infrastructure in straight.el; the system is already extensible enough.