Add an API for processing packages

[jlfranklin]

Based on a conversation between me and @docwilmot on Gitter, plus some additional brainstorming:

When a contrib module release is cut, there are a number of things that need to be done. The most basic are:

• Add tags to the info file with the module's version, project machine name, and release timestamp
• Update the XML that core's updater and project installer use to manage packages
• Zip up the final package and send it to the package repository (Github, in this case.)

There are some additional things we want to add to the packages:

• Collect the name(s) of the maintainer(s) so their backdrop.org profile(s) can list all the packages they maintain
• Pull screenshots from a specified directory. (backdrop-ops/backdropcms.org/issues/487)
• Add a digital signature (backdrop/backdrop-issues#1992)

The existing process is started by a webhook in Github invoking a webservice in backdropcms.org. The Project Github module services the webhook in _project_github_create_package(), performs all the required packaging steps itself, and pushes the final zip file back to Github.

While Project Github is the one used by Backdrop (today), it should be possible to trigger the packaging process from a Gitlab webhook or a simple Git commit hook, or some other external source.

Conceptually, Project should be the real driver of the packaging process, invoking hooks to allow other modules to contribute their thing.

To accomplish the above goals, the following was proposed:

• Add a project_do_package() function (replacing do with something less vague.) That function is called by a release trigger (like Github's _project_github_create_package()), and calls project_preprocess_package() and project_postprocess_package() to do early (tagging) and late (zipping) package processing.
• Move the tagging of the modules from project_github to project as it is something done regardless of the git repo and package repo locations.
• Both of the pre/post process functions will invoke hooks to allow other modules to do their own package manipulation.
• The preprocess hooks should work on the module files (.info, .module, .admin.inc, etc.), the postprocess should work on the finished zip file.
• A separate call after or near the end of project_postprocess_package() that generates the XML (see: the current project_cron() call.)
• Create a Project Screenshot to extract the images from the module.
• Include the Project and Project Release nodes in the hook calls.

Potential issues:

• Some processing must run after all other modules have finished changing the package, like creating digital signatures. Will calling the modules in weight order be sufficient or will we need to implement something like how Input Filters orders filters to set up a pipeline?
• Separate source (git repo) and sink (package repo) modules? Would we want to Github to trigger the process, but save the end package in an S3 bucket? Or would that be done by an Project S3 module's postprocess hook and a second copy goes to Github anyway?
• Is there an existing API that we should mimic? Input Filters? Node API? Could we invoke the hook_project_pre/postprocess_package() hooks from node's hook_insert() when creating the Project and/or Project Release nodes?

[quicksketch]

The packaging system right now is a little mysterious where things happen, so arranging the code so that things are in more expected locations sounds generally good. I think it's surprising to me ever time I go hunting for the packager code and find it in the project_github module.

What sort of data structure do you envision passing around as the process of packaging takes place? Looping over files in the filesystem is not a particularly easy bit of code and doing it multiple times (e.g. to modify .info files and to pull out images) seems like something we'd want to avoid.

I don't think we need to overcomplicate things as far as determining execution order. There's no need for a UI or configurable ordering, this code will be run (almost) exclusively on backdropcms.org, I think we should make it as direct as possible for that purpose. So overall I think module weights and hooks should be fine and hook_module_implements_alter() can be used if need to change the order.

I'm not sure what you're suggesting we use S3 for. We could start putting files there, but thus far it seems like attaching the files to the GitHub releases has worked well and we get convenience benefits like the archive being accessible from the GitHub release page, automatic download tracking, and free hosting.

[jlfranklin]

I think it's surprising to me every time I go hunting for the packager code and find it in the project_github module.

When I found that, I thought, "huh, I wouldn't have put this here, except I probably would because this module is used in exactly one site and it just needs to get done. Why build three modules when I can do it all in one function?"

What sort of data structure do you envision passing around as the process of packaging takes place?

I left that part deliberately vague, partly because I wanted to hear other people's ideas, and partly because I figured it would depend on how the process is built.

I've thought about doing everything in hook_node_presave(), hook_insert(), and hook_node_insert() as the Project Release nodes are created, and using the node object as the data structure passed around. The more I think about it, the more I don't like that solution. It avoids creating yet-another-API, but by using Node API in ways it was never intended and with a lot of side-effects.

The next most common pattern in Backdrop is for project_do_process() to create a tagged array of stuff and pass it around. The initial array would have the list of files, the path of the temporary directory where they're extracted, the .info file parsed out, and an array or two that will become the XML metadata. Should we include the Project and/or Project Release nodes in this array? Will project_do_process() create the new Project Release node automatically?

The screenshot module might add the images to the array in an early hook. The digital signatures are added in a late postprocess hook. Project Github adds download links to the Project Release node and XML. Somewhere in there, the .info gets reconstituted, the final module archive is created, the XML is written out, and the Project Release node is saved.

I don't think we need to overcomplicate things as far as determining execution order. [...] and hook_module_implements_alter() can be used [...]

I've never even heard of that hook before now. (What's the emoji for "lucky 10k?")

If we don't need it, then we don't need it. Given what we have in place today and features we want on the horizon (.info tagging, screenshots, digital signatures), it's already clear that processing order will matter and so we have to keep it in mind. In the end, we may just need to add an extra hook or two to project_do_process().

I'm not sure what you're suggesting we use S3 for.

Project defines a pipeline where code is pulled from somewhere (source), modified to make it a release artifact (process), and then put someplace Backdrop sites and site builders can find it (sink.) Right now, nearly everything in that source -> process -> sink pipeline happens in project_gitlab, because that meets Backdrop's needs.

If this were a more generic pipeline, built for more than one (or two) sites, then there would be no assumption that source and sink are the same place and managed by the same module. The mention of S3 as a potential sink is just a plausible example of a separate sink, not a suggestion for how Backdrop should distribute modules.

[quicksketch]

@jlfranklin I merged in https://github.com/backdrop-contrib/project/issues/37 that I think will accomplish the ability to do a some of these tasks (e.g. remove screenshots, modify .info files), but leaves most of the code in place and continues to tie it to the project_github module. I'm definitely open to more comprehensive approaches but I'll be interested to see what situations aren't covered by that approach, where the files about to be included in the archive are just passed around by reference.

[jlfranklin]

The patch in #37 is as clean as it gets. If we need more, we can add more later.

[quicksketch]

Great, let's close this one up as a duplicate of https://github.com/backdrop-contrib/project/issues/37.