Update all drake-*.csail.mit.edu sites to use Certbot

[BetsyMcPhail]

drake-jenkins.csail.mit.edu and drake-cdash.csail.mit.edu both recently updated to use Certbot. This process was very simple as we could log directly into the Linux AWS images.

The following sites all still use InCommon certificates which must be updated annually by a manual process.

drake-homebrew.csail.mit.edu drake-mirror.csail.mit.edu drake-packages.csail.mit.edu drake-apt.csail.mit.edu

drake-girder.csail.mit.edu

The first 4 servers are AWS CloudFront distributions and girder is an AWS Elastic Beanstalk application.

This issue is to investigate how to use Certbot with the AWS infrastructure.

[svenevs]

Noting that I'm not a web person, from what I can tell,

• CloudFront options:
  • Setup a dedicated IAM user for updating CloudFront. It appears you will need a dedicated server to do this action automatically, see this article for an example. Essentially, have some other server generate new certificates and then upload them to CloudFront.
    • Could base it off of the certbot docker image as well.
    • Referring article to the previous.
• Elastic Beanstalk
  • See this article, it seems possible.
    • NOTE: they setup a cron job as part of the deploy that renews the certificates twice daily. That seems excessive, but I'm not familiar with how Elastic Beanstalk actually works. Certbot recomends every ~60days.

[BetsyMcPhail]

Note to myself, drake-apt.csail.mit.edu expires in May and the others expire in October, let's get this done in time for the May renewal.

[BetsyMcPhail]

The certificate for drake-apt.csail.mit.edu has been updated. Let's Encrypt certificates are only valid for 90 days (https://letsencrypt.org/docs/faq/#:~:text=Our%20certificates%20are%20valid%20for,your%20certificates%20every%2060%20days), so the next step is write a simple script to automatically do the renewal.

[BetsyMcPhail]

There are more detailed instructions in the internal CI documentation, but the general process is to:

1. Use certbot to create/renew a certificate using the http challenge (can be done locally or from an EC2 instance, for example)
2. Add/Renew the certificate in the AWS Certificate manager
3. Add/Refresh the certificate on the CloudFront instance

All of the steps can be scripted.

drake-apt.csail.mit.edu expires on Aug 9, 2023. There is a cronjob setup on an EC2 instance to renew this certificate every-other-month on the 11th. I will check on July 11 to make sure the certificate renewed successfully. If everything looks good, we can go ahead and switch the remaining sites over to this process.

Edit: I'm going to move this issue out of "In Progress", I have a slack reminder set-up for the July renewal.

[BetsyMcPhail]

The cronjob ran last night but had some AWS access issues. I fixed those up and completed the update by hand. I also improved the scripts and updated the documentation.

drake-apt.csail.mit.edu now expires on Oct 9, 2023 and should be renewed on Sept 11, 2023. I have a slack reminder setup to ensure that the auto-update went smoothly.

The other certificates expire in October so I am going to make sure we have the process straightened out before switching them over.

[BetsyMcPhail]

drake-apt.csail.mit.edu auto-renewed as expected on Sept. 11. The remainder of the CloudFront instances (drake-homebrew.csail.mit.edu, drake-mirror.csail.mit.edu, drake-packages.csail.mit.edu) have been renewed manually. All of these expire in December are set to auto-renew on November 11.

[BetsyMcPhail]

https://drake-girder.csail.mit.edu/ has also been updated.

[BetsyMcPhail]

(Finally) closing this issue! Everything has been converted to Certbot or will be handled separately.

[ggould-tri]

This new cert does not appear to work with tools such as curl

ggould@baymax:~$ curl --verbose https://drake-girder.csail.mit.edu/
*   Trying 52.73.240.167:443...
* Connected to drake-girder.csail.mit.edu (52.73.240.167) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Unknown (21):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

and python requests

>>> import requests
>>> requests.get('https://drake-girder.csail.mit.edu/')
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.10/ssl.py", line 1071, in _create
    self.do_handshake()
  File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)

(both of the above under ubuntu Jammy)

I wonder if this is a missing intermediate cert for your CA, InCommon RSA Server CA?

[BetsyMcPhail]

The girder certificate is working with curl now.