Stabilize the wpt-live cert renewal process

[jcscottiii]

There have been multiple instances of when the certificate renewal fails as reported in #57.

This issue serves to track the work to do if we want to re-work it to stabilize the process.

Option 1 is the preferred option. But option 2 is available as a short term option.

Option 1: Use a Load Balancer and let the load balancer automatically renew the certificate

Currently, there is a TCP load balancer used (not in the diagram on the README page). Cert renewals have to be managed by the team as a result when using a TCP load balancer. There are now HTTPS load balancers that renew the certificate automatically.

Docs: https://cloud.google.com/load-balancing/docs/https/setup-global-ext-https-serverless

*Google Managed Certificates*                               *GitHub*
      |                                                     |
[TLS certificate auto renewed]                      [WPT source code]
      |                                   .------------.    |
      V                               .-->| wpt server |<---+
.--------------.                      |   '------------'    |
| GCP HTTPS LB |---------------------->                     |
'--------------'                      |   .------------.    |
                                      '-->| wpt server |<---'
                                          '------------'
Legend
                    .---.               +++++
*   * external      |   | GCE           +   + object     [   ] message
      service       '---' instance      +++++ store            contents

Steps:

• Modify the terraform to deploy a HTTPS load balancer
• Modify the terraform to have the cloud dns point to the IP address of the HTTPS load balancer. Verify that the cert is now issued by Google instead of Lets Encrypt.
• Modify the code to stop pulling the certificate from the cert store
• Modify the terraform to remove the existing TCP load balancer and the cert store bucket
• Remove the cert renewal Dockerfile and code from the repository.

Pros:

• One less process to worry about
• Currently the load balancer is set up for a single region. Doing this work will allow us to configure it for multiple regions (we don't need to deploy to multiple regions yet). That way, if a region goes down, we can easily deploy to another region.

Cons:

• Relatively a lot of work compared to option 2.

Option 2 - Migrate to Cert Renewal Process to a cron job in Cloud Run

The Cert Renewal Process is already in a Dockerfile. This makes it easy to convert to Cloud Run which can run Docker images that run on a schedule. Docs.

*Let's Encrypt*                                         *GitHub*
      |                                                     |
[TLS certificate]                                   [WPT source code]
      |                                   .------------.    |
      V                               .-->| wpt server |<---+
.--------------.    +++++++++++++++   |   '------------'    |
| cert-renewer |--->+ certificate +---+                     |
'--------------'    +    store    +   |   .------------.    |
                    +++++++++++++++   '-->| wpt server |<---'
                                          '------------'
Legend
                    .---.               +++++
*   * external      |   | GCE           +   + object     [   ] message
      service       '---' instance      +++++ store            contents

*Note It's the same architecture as the current one in the README

Steps:

• Modify the terraform to add a cloud run instance and run it on a schedule
• Modify the Docker image to not run in a loop anymore.
• Rebuild the images and deploy

Pros:

• Relatively quick to do
• Currently, it seems like the first renewal works but afterwards it fails in the loop. Since we are now changing this to run one time (but in a schedule controlled by cloud run), we should get that same "first" working renewal every time it runs.
• Compared to the existing solution, it should be less expensive. Currently the app engine process is always on but the process only runs once a day. That means it is up and idle for a long time. Option 2 will run it only on the schedule

Cons:

• Still need to monitor this external process
• Still not resilient to the single region going down.