Create a debian package hosting mirror with Fastly in front with long cache time

[renoirb]

Estimate

• 2h

  Summary:

In order to have a reliable system, we need to host our .deb (a.k.a. apt-get source location) files from a web server. But what if the underlying web server is down too? Let’s leverage Fastly by configuring it to store the file for long period of time.

Current setup is that we copy contents of folder on every servers. It would be more efficient if we had it available from the web directly.

Tasks

• [x] Setup domain name zone file (e.g. static.webplatform.org).
• [x] Pull files directly from DreamObjects bucket.
• [x] Setup a debian repository folder as namespace (e.g. /wpd/packages/apt/)
• [x] Setup basic APT repo, see Packages.gz.
• [x] Ensure apt repo ONLY allow to download files that are explicitly known.
• [x] Setup all nodes to rely on it
• [x] Configure Fastly/Varnish to keep it for a very long time

  Related

• Create a public download archive stored directly in DreamObjects (incomplete)

[renoirb]

Fold into upcoming talk as an example.

[renoirb]

Related webplatform/ops#121 ?

[renoirb]

Done, and it works.

[renoirb]

Issue we had was because apt would make requests to http://static.webplatform.org/wpd/packages/apt//file.deb and Varnish didn't like the double-slashe ("//"). Fixed it like that