Consolidate template repositories

What is the problem or limitation you are having?

Each cookiecutter template for Briefcase currently has its own repository....but these repositories are remarkably similar and could potentially benefit from consolidating in to a single repository.

Describe the solution you'd like

With the release of cookiecutter v2.2.0, the ability to nest multiple templates within a single repository became a reality. Therefore, we are no longer technically limited to individual repositories.

One motivation for this is maintenance simplification. Twice recently, we had to go through every single template repo and create an identical PR to apply the same update to each one. The similar and shared nature of these repos strongly suggests to me they benefit more from sharing a repo than using their own.

One foreseeable complication is briefcase-windows-app-template and briefcase-macOS-app-template since these build and contain their stub binary. Given that briefcase-macOS-app-template has already ballooned to 221MB, this binary eventually needs to live outside of the repo; this would provide a convenient opportunity to do this.

I know a desire to ship the templates as packages has been expressed; although, as also expressed, using a consolidated repo for the templates may make even more sense at that point. ~(I'm not sure I'd like to go so far as putting the templates in the Briefcase repo, though....that's gonna complicate CI quite a bit.)~

Describe alternatives you've considered

Status quo.

Additional context

I created this meta-issue to formalize this discussion and ideally create a rough sketch/design of the needed work. Before that, though, I'd like to capture the constraints and considerations for this move.....so, please comment them if you've got em.

Twice recently, we had to go through every single template repo and create an identical PR to apply the same update to each one. The similar and shared nature of these repos strongly suggests to me they benefit more from sharing a repo than using their own.

Agreed.

One foreseeable complication is briefcase-windows-app-template and briefcase-macOS-app-template since these build and contain their stub binary. Given that briefcase-macOS-app-template has already ballooned to 221MB, this binary eventually needs to live outside of the repo; this would provide a convenient opportunity to do this.

I see that the macOS stubs are over 7 MB each, while the WIndows stubs are only 130 KB. I assume this is because Windows uses a dynamically linked libpython. @freakboy3742 is currently looking into doing the same on macOS, which should make it unnecessary to deal with the complexity of storing the stubs separately from the template.

However, doing a shallow clone would probably still be a good idea.

I'm not sure I'd like to go so far as putting the templates in the Briefcase repo, though....that's gonna complicate CI quite a bit.

In what way? Wouldn't this make it easier to test Briefcase and template changes together? Right now we have a way for a template PR to specify a corresponding Briefcase PR to test against, but not vice versa.

I'm not sure I'd like to go so far as putting the templates in the Briefcase repo, though....that's gonna complicate CI quite a bit.

In what way? Wouldn't this make it easier to test Briefcase and template changes together? Right now we have a way for a template PR to specify a corresponding Briefcase PR to test against, but not vice versa.

Yeah, you're right, I take that back...especially since testing dependent changes would be much easier then.

I was thinking the CI for the templates and Briefcase wouldn't mix well and we'd have to conditionally trigger different CI based on what's changing or something....but in reality, running Briefcase's CI tests all the templates already....so, anyway.

Twice recently, we had to go through every single template repo and create an identical PR to apply the same update to each one. The similar and shared nature of these repos strongly suggests to me they benefit more from sharing a repo than using their own.

Agreed.

Also agreed.

One foreseeable complication is briefcase-windows-app-template and briefcase-macOS-app-template since these build and contain their stub binary. Given that briefcase-macOS-app-template has already ballooned to 221MB, this binary eventually needs to live outside of the repo; this would provide a convenient opportunity to do this.

I see that the macOS stubs are over 7 MB each, while the WIndows stubs are only 130 KB. I assume this is because Windows uses a dynamically linked libpython. @freakboy3742 is currently looking into doing the same on macOS, which should make it unnecessary to deal with the complexity of storing the stubs separately from the template.

However, doing a shallow clone would probably still be a good idea.

Another approach - don't put binaries in the template at all. We could treat them as a support artefact, similar to the support package itself. We could then push an update to the stub binary independent of the template, and vice versa. Binaries aren't well suited to being stored in git to start with, so avoiding the problem entirely would seem appropriate.

I'm not sure I'd like to go so far as putting the templates in the Briefcase repo, though....that's gonna complicate CI quite a bit.

In what way? Wouldn't this make it easier to test Briefcase and template changes together? Right now we have a way for a template PR to specify a corresponding Briefcase PR to test against, but not vice versa.

I'd like to propose a slightly different approach.

Let's make Briefcase template pip-installable packages, with the cookiecutter payload being just one part of the content that a template provides. Essentially, the entire template would a plugin against a known interface, and the location of the template content becomes just one of many potential API endpoints that the "template package" can satisfy.

This also means that the macOS/Windows app templates can internally determine how they obtain their stub binaries. The template would expose a "roll yourself out" API endpoint, and the implementation of those packages would download and install the stub as required.

This would be reasonably well suited to integration into the Briefcase repo itself - it just becomes another wheel artefact produced by the build, installed locally as an editable package for testing purposes.

It also means we can rely on pip version resolution rules, rather than having our own bespoke "epoch" rules. Templates are just packages, and they're dependent on specific versions, or version ranges, using pip specifiers and resolution rules.

Wouldn't that make it much more complex to use a custom template? Right now, a template can be any directory location laid out in a cookiecutter-compatible format, and any custom behavior can be done using a hook script, just as the macOS-app and windows-app templates currently do. So I'm not sure what we'd gain by making them installable packages.

I guess it depends how we manage the template projects. If they're in repo, then yes - it will be a little more complex, because an end user would need to fork the briefcase repo and then install one subbranch of that repo. That's entirely possible, but a lot less obvious than setting a template URL.

That said - the same problem exists if we merge all the templates into the Briefcase repo. Right now, if I want to customise the macOS-XCode template, I can fork that template. If the templates are all in the Briefcase repo, then I need to fork the entire Briefcase repo.

FWIW: The main benefits I see are that we are able to expose (and control) a clear programmatic interface to template generation, rather than relying on the command-line interface of cookiecutter as the interface; and we get to leverage Python's version dependency management rather than generating our own bespoke version compatibility tools.

This discussion has turned into three orthogonal proposals:

Get binaries out of the templates and manage them as support artefacts
Merge the templates into a single repo (either the Briefcase repo, or a single template repor)
Modify the templates to be Python packages.

I think (1) is a no-brainer - templates are the wrong place for binaries.

(2) was motivated by the recent "bumping a version becomes 9 separate PRs" change. I agree that's a lot of busy work (which is what my original agreement was reflecting). However, it's not work we have to do that often, and merging into a single repo also means we lose the end-user fidelity of working on just the template that an individual user cares about (i.e., why do I need to clone the Android template if I want to tweak the macOS app deployment?).

I suggested (3) on the basis that if we're going to do (2), we might as well make a clean break and leverage Python's package plugin interface. However, it could be done independent of (2). It might open up some options for some interesting end-user customisation... but I don't have any specific plans for what that might look like.

So - I'm +1 on (1), but -0 on (2) and (3).

beeware / briefcase