web-platform-tests / wpt.live

A live version of the web-platform-tests project
https://wpt.live/
15 stars 11 forks source link

Report: service outage due to change in set of available origins #12

Closed jugglinmike closed 6 years ago

jugglinmike commented 6 years ago

Today, WPT was extended with support for more origins. This project was originally built to treat the available subdomains as system configuration. That is: it would reference the instructions provided by WPT, but only at deployment time. We update and restart WPT far more often than we deploy the configuration. Because the server asserts proper system configuration at start-up time, changes like today's were incompatible with web-platform-test.live's underlying assumptions. As a result, the server went down immediately following acceptance of the aforementioned patch (roughly 16:32 UTC).

The set of supported subdomains is liable to change with any commit, making it less like system configuration and more like application input. The system needs to be able to update its "hosts" file dynamically each time it receives a new revision of WPT.

Fortunately, this is a problem we have already solved for the results collector. I was able to re-use that solution here, and as a result, the project now keeps the "hosts" file up-to-date for every WPT revision it attempts to run:

https://github.com/bocoup/web-platform-tests.live/commit/fd926a8b38605a469db1f96598fcc3eca18196d3

This also required reconfiguring the project's DNS settings to support "wildcard" CNAME record. That was implemented via a commit to Bocoup's repository for web-platform infrastructure:

https://github.com/bocoup/infrastructure-web-platform/commit/111587c777e4e0569241a3fab22b5a5a3e83ab1a

The system has been operating as expected following the deployment of those patches.

jugglinmike commented 6 years ago

Actually, the system hosts file is necessary only for internal routing during development. It's not relevant in production contexts. This issue was resolved by the DNS change alone. I've removed the hosts file management logic to simplify the system.