Automate uploading of stable ISOs

cassidyjames commented 3 years ago

Problem

Right now we can kick off a stable ISO build in the OS repo, and it gets uploaded to DO. However, the process of getting that ISO up on the CDN and available to users is manual, and only @lewisgoddard can currently do it. This creates a human bus factor that may become more apparent as we resume our monthly cadence of ISO refreshes.

Proposal

@btkostner and I talked briefly (and he has more thoughts!) about how that last step could be automated. He mentioned something like an RSS feed from the builds site to seed @lewisgoddard's CDN boxes. I'm sure he can share more. :)

Prior Art

btkostner commented 3 years ago

The idea here being that our download servers can pull the RSS feed, check if it has the latest version, if not, use an api endpoint to download them (or have keys to generate the signed url themselves, not determined yet), and then do the normal dl setup. Eventually we can then hook up the front end to auto pull as well, and we have a much more automated CI build to homepage download pipeline.

We could also have the RSS feed (or a specific torrent feed) be used for seeders to auto seed our download torrents

lewisgoddard commented 3 years ago

Current State

Current builds aren't uploaded per-se. The torrent reference is added and the download infrastructure pulls down the ISO from the swarm, verifying each chunk. This isn't 100% automatic, but would continue to work so long as at least 1 person is seeding.

Issues

How do we approve builds to go to infrastructure? Manually adding items to the RSS?

Right now the problem is:

We want to automatically add torrents to infrastructure.
We don't want to push every build to infrastructure, even if they're marked as stable.
We want to remove old ones to avoid running out of space (we can fit at least 5, but builds are getting bigger).

1-change publishing

We want to publish builds with only 1 change. The RSS feed is the right way to do that.

Add an RSS feed that contains ONLY the builds you wish to seed, probably the current (proposed) stable, previous stable, and previous major releases.

Add a key to access the RSS feed that appends signed-url webseeds to the magnet links.
OR
Make sure at least 1 person seeds the full ISO to the swarm.

Use programmatic access to infrastructure that:

Adds new torrents as they appear on RSS.
Removes old torrents as they disappear from RSS.

The same programmatic access can be used to make a somewhat public dashboard for each download server as a status page.

Format

There is a standard for that. https://www.bittorrent.org/beps/bep_0036.html

We should include the torrent file as spec states, but I'd prefer to use magnet on the infra as we don't have to worry about the download borking.

elementary / builds