Confused about mirroring of GNU ELPA packages

[aspiers]

Apologies in advance if this issue poses stupid questions. I tried to understand all the mirroring, mirroring of mirrors, and mirroring of mirrors of mirrors described in the README, but I ended up with my head spinning and feeling no less confused.

The main thing I am trying to understand is: why does my ~/.emacs.d/straight/repos/undo-tree git repository have "neutered" history, rather than being a direct clone from the official upstream repository http://www.dr-qubit.org/git/undo-tree.git (as linked to from the package's homepage)? By neutered history, I mean that it looks like this:

Two follow-on questions spring to mind:

• Where is the true history getting lost? I note the following:
  • The undo-tree branch of emacs/elpa.git on Savannah has the full history.
  • https://github.com/emacsmirror/undo-tree has the full history.
  • https://github.com/emacs-straight/undo-tree does not have the full history.
• Why is straight.el cloning from a mirror rather than the true upstream source repo? Is it because undo-tree is only "registered" in GNU ELPA and not MELPA, so straight.el doesn't know about the upstream source repo?

Of course I appreciate that I can manually specify a recipe which points to the upstream repo. But IMHO it would be better if straight.el did this automatically for packages like undo-tree where presumably it can make more sense to get it from the upstream. At least it does in the cases where the user wants the latest master in order to be able to develop on it and contribute patches back. Maybe GNU ELPA offers some extra level of quality assurance and stability, but even if so, not all straight.el users will want that 100% of the time.

I may well have misunderstood some things, but bearing in mind the above, I have two humble suggestions:

• Add a diagram to the documentation showing the "flow" of package code from the various upstreams through the various mirrors.
• Provide some strategy, if it doesn't already exist, for dealing with packages like undo-tree, where there is an upstream repo available which can be preferable to getting it from GNU ELPA.

Thanks!

[raxod502]

I do my best to answer. There is currently no documentation covering these topics, unfortunately. It should be written. Feel free to ask follow-up questions if there is something still not clear.

Why is straight.el cloning from a mirror rather than the true upstream source repo?

• undo-tree isn't listed in MELPA, only in GNU ELPA, so that's where straight.el gets it
• the build system of GNU ELPA is set up in a way that makes it basically impractical to use for a source-based package manager, so (as per the default non-nil value of straight-recipes-gnu-elpa-use-mirror) we download from a mirror
• you can see the source code for the mirror at https://github.com/raxod502/gnu-elpa-mirror
• the way mirroring works for GNU ELPA, we just run the build system for GNU ELPA and then copy all the files out from each package, and commit any changes (destroying all meaningful history)
• this is because GNU ELPA does not follow a standard layout like MELPA, some packages are vendored, others are pulled out of Emacs core, some are mirrored from external Git repositories, etc.
• we could update GNU ELPA Mirror so that it detects when a package in GNU ELPA is directly mirrored from an external Git repository, and in this special case just mirror the whole repository directly, but I didn't bother to do this yet (if we make this change then we should move the current history on such repositories to an archival branch, so that the commits still exist for current lockfiles)

https://github.com/emacsmirror/undo-tree has the full history.

Because Jonas did a better job than me in this case. Note that the emacs-straight mirror is a mirror of GNU ELPA, not a mirror of Emacsmirror. Only epkgs is mirrored from Emacsmirror, the actual packages in Emacsmirror are not mirrored. You could get the full history by (straight-use-package '(undo-tree :source emacsmirror-mirror)) because that would pull from Emacsmirror rather than the GNU ELPA mirror.

Add a diagram to the documentation showing the "flow" of package code from the various upstreams through the various mirrors.

Great idea, that would help a lot. Currently, there is effectively no documentation on GNU ELPA Mirror. This should assuredly be fixed.

Provide some strategy, if it doesn't already exist, for dealing with packages like undo-tree, where there is an upstream repo available which can be preferable to getting it from GNU ELPA.

This would certainly be desired as well, and should be covered by the last bullet point above.

[aspiers]

Thanks a lot for this quick and extremely helpful response! That all makes a lot of sense, especially this:

we could update GNU ELPA Mirror so that it detects when a package in GNU ELPA is directly mirrored from an external Git repository, and in this special case just mirror the whole repository directly, but I didn't bother to do this yet (if we make this change then we should move the current history on such repositories to an archival branch, so that the commits still exist for current lockfiles)