Massive refactor

[tarsius]

Subsequently I am criticizing the existing code quiet harshly. I am pretty sure that @purcell and @milkypostman will understand that correctly, but others might read this as well, so I would like to emphasize that this is not intended as a criticism of the authors or their skills. I maintain a project similar to Melpa and as part of that maintain tools similar to this one, and know how much work it is to just keep things going. It is very tempting and often the only course of action that won't make you burn out or otherwise go insane to just keep adding features and fixing bugs without taking a step back and refactor.

Motivation

1. A long time ago I have said that I would like to replace the timestamps that are used on Melpa (non-stable) with version strings of the form VERSION.UPDATE-COUNT. Or something like that. I think that the plan outlined in https://github.com/melpa/melpa/issues/2955 is fairly sound, but I haven't looked at it in a while.

   Many others have expressed interest in something like this but nobody has implemented anything. Anyway the point is that I think it should be fairly simple to do that. But when I went looking for how to inject that feature into the existing code I ran into a lot of accidental complexity.

2. I am the maintain the Emacsmirror, which in many ways is similar to Melpa. The primary output of Melpa is a collection of package.el-compatible packages, while the primary output of the Emacsmirror is a database containing a list of Git repositories containing Emacs packages, as well as lots of metadata about these packages.

   For most users the output of Melpa is more useful, but I am happy that there now are a few users who use my Borg package manager and my Epkg manager.

   But my work on the Emacsmirror benefits Melpa users as well. In the early days of the Emacsmirror I went looking for packages to add myself. Nowadays I rely on Melpa for that purpose. Some time after a package is added to Melpa, I also semi-automatically add it to the Emacsmirror.

   Because the Emacsmirror extracts more metadata about packages than Melpa does, that often leads to the discovery of issues that @purcell did not catch during his (very valuable) review. I then contact the maintainer and ask them to fix the issues, or even do it for them. These fixes make it into Melpa as well on the next update.

   Additionally, even though I don't use package.el myself, I am a top contributor to Melpa (right after the two head honchos). By comparing the data found in Melpa and the Emacsmirror I not only find issues in individual packages but also to issues in Melpa recipes and that leads to a lot of commits.

   Anyway the Emacsmirror extracts a lot more metadata about packages than Melpa (or that package.el currently supports (give epkg a try)) and some of that could be ported to Melpa (and package.el). But again the accidental complexity gets in the way.

   The Emacsmirror too could benefit from Melpa more if package-build.el were more similar to the respective tools used to maintain the Emacsmirror.

3. The reason I am doing this now is that after the next release Magit's development will move to two long-lived branches (likely maint and master). I would like Melpa Stable to continue to contain releases such as 2.12.1, Melpa to stay on maint on offer versions like 2.12.1.5, and my own Elpa archive to offer versions like 2.99.101 from master. For that I need VERSION.UPDATE-COUNT support, as well as some other things that I don't want to implement on top of the current package-build.el implementation.

Melpa could be improved in many ways, and many users request such changes, but when they are told by the overworked maintainers to lend a hand not much happens. I think this is partially due to the accidental complexity that makes it hard to add something new. The primary goal of these commits is to remove some of that complexity so that package-build.el and by extension Melpa can grow new features.

Status

• I have only lightly tested these changes and I guess I should recommend that this pull-request isn't merged until we have done some more testing.

  Similar pull-requests that I created in the past tended to have some bugs. These bugs were rather silly and "easily preventable", but that wasn't only the result of too little testing but is also to be expected when replacing accidental complexity with a cleaner implementation. When there are many unnecessary complications, then it is easy to mistake a complication that actually is necessary for one of the unnecessary ones.

  So maybe it wouldn't be a bad idea to merge this pretty soon anyway, but on a day where we all are available to quickly fix any regressions encountered by users.

• This is only a step toward a cleaner code base.

  There are some temporary hacks. I have removed many obsolete doc-strings without replacing them with new ones, because I only have so much motivation to improve things incrementally. I am doing it for the code, but the documentation I don't want to update until things are done instead of over and over again.

  Also I have to turn my attention to other things now. So it could be a while until the next patch series.

Changes

• Make code that determines the appropriate version self-contained.

• Use fewer arguments to pass data around.

• Use Eieio objects to represent recipes instead of plists. These objects contain more information than the old plists, which is what makes it possible to use fewer arguments to pass information around.

• Replace leaky abstractions with self-contained functions, doing one thing only, and all of it.

  This too is made easier by the use of objects that contain all the necessary information with no overhead such as having to add additional arguments to many functions just so that the work can be done where it makes most sense to do it. Previously some work was done in the wrong place, simply because that was the only place where the required information was available.

• Organized the file into logical units by using sections. This was a very necessary step, which I already began in earlier patch series. It helps a lot recognizing accidental complexity and removing it.

  This required moving some existing function definitions. The order of the sections themselves and the order of definitions within a given section is still fairly random.

• Remove many helper functions, by inlining them. These helper functions often existed because the functions that called them were to complicated, making it necessary to move at least some of the code elsewhere. But after cleaning up the calling function that often wasn't necessary anymore.

  The existence of many tiny helper functions that are defined in weird places and have sub-optimal names and doc-strings, makes it very hard to fully grasp what a certain task involved.

  Having everything in one place helps a lot here. Which isn't to say that tiny helper functions are bad per se, but at this point in the evolution of the code base, I feel they do more harm than good. Also they often existed to cover up another issue, not to increase the separation of concerns.

This patch series does not fix all of the mentioned issues. It fixes some instances of each of the mentioned issue categories, while leaving others untouched.

[tarsius]

I'm consequently not sure how best to fully assess & communicate the likely downstream effects not just on MELPA but on Cask, Quelpa etc., and any other code that uses this.

We could just use it for Melpa for a while.

[purcell]

We could just use it for Melpa for a while.

Sounds like a plan!