Protect sites by TLS.

[rivernews]

Steps

• [x] Create cert manager
• [x] Run CRD command
• [x] Create cluster issuer
• [x] Modify ingress resources w/ the tls block

Ref

• A Latest Cert-Manager quickstart
• B (May 2019) Digitalocean: How To Set Up an Nginx Ingress on DigitalOcean Kubernetes Using Helm
• C (Feb 2019) SO: Running CRD command in terraform, success story
• D (Feb 2019) Personal blog: Setting up TLS by terraform on AKS.
• E (May 2019), Medium, Cert manager + external DNS, v0.7.

Other Ref

• Github issue discussion about stable/cert-manager not working, latest version discussion included.

[rivernews]

Exec

1. Scaffold: D
   • Caveats:
     • Helm release cert-manager: pins to v0.5.2 due to fast-changing spec of cert-manager
     • Does not include CRD command, but uses a custom cluster issuer chart in his repo.
2. Base: B
   • Should be pretty thorough.
3. Adapt C
   • Includes the CRD command explicitly, and in Terraform, which is nice.
   • Seems to work according to her response.
4. Latest A
   • Has latest version.

---

Ready to exec.

Concerns

• Some are non-reversable, like kubectl create ClusterIssuer. Best to have a real terraform resource handle this, like helm_release.

  • There's a blog post about using helm_release to install the cluster issuer. Is this possible for latest version of cert-manager v0.9.1 as well?

• OK, seems like it may be OK to just run terraform

  • provisioners by default only runs upon creation, not modified
  • In case we really need to destroy i.e., CRD, can refer to this AKS TF module. It basically run an additional kuberctl delete CRD command.

[rivernews]

Debug

• By destroy and re-apply cert-manager and the ingress resources, also delete the cluster issuer's secret, then change the issuer's name to letsencrypt-prod (exact, not appending anything else), plus don't specify a namespace in ClusterIssuer yaml, we got Events in the certificate (following along this github issue):

Events:
  Type     Reason          Age                  From          Message
  ----     ------          ----                 ----          -------
  Warning  IssuerNotReady  2m2s (x2 over 2m2s)  cert-manager  Issuer letsencrypt-prod not ready
  Normal   OrderCreated    2m1s                 cert-manager  Created Order resource "letsencrypt-prod-secret-1287490335"
  Normal   OrderComplete   94s                  cert-manager  Order "letsencrypt-prod-secret-1287490335" completed successfully
  Normal   CertIssued      94s                  cert-manager  Certificate issued successfully

[rivernews]

OK, almost there.

Chrome will not show security error page now. But instead, has a "not secure" badget besides.

---

UPDATE

After waiting for a while, it's now working and Chrome shows it as secure! 🎉

[rivernews]

Wildcard?

The key is to change from using http01 challenge, to dns01 challenge. Because a wildcard certificate requires dns01.

The next thing that would be cool is to set a wildcard TLS. However, this is optional, and it might take a lot of work, time and effort.

Checking rate limit for letsencrypt

See the cert issued for rx name.

Using Route53 as DNS01 Challenge provider

• Turns out that this is the key, you have to set the ingressShim on cert-manager.
• Also inspired by route53 settings in this Medium post and that Medium post.

Make the changes and debug

• [x] Follow doc to flush all TLS related resources
• [x] Change ingress resource tls to wildcard
• [x] Change challenge in clusterissuer to dns01
• [x] Apply
• [x] Verify
  • [x] Check ingress annotation, using staging?
  • [x] Check clusterissuer
  • [x] Check certificate, successfully created?
  • [x] Visit website using https, shows Fake Intermediate or something like Fake LE Intermediate X1?
• [x] Ready to switch to prod
  • [x] Flush all TLS related resources
  • [x] Change to prod letsencrypt endpoint
  • [x] Apply
  • [x] Repeat "Verify" above, while changing to prod.

[rivernews]

🎉 Certificate success with DNS challenge and wildcard domain!

Using letencrypt prod w/ DNS challenge. K8 certificate shows:

Events:
  Type     Reason          Age                  From          Message
  ----     ------          ----                 ----          -------
  Warning  IssuerNotReady  4m1s (x2 over 4m1s)  cert-manager  Issuer letsencrypt-prod not ready
  Normal   OrderCreated    4m                   cert-manager  Created Order resource "letsencrypt-prod-secret-649678329"
  Normal   OrderComplete   4s                   cert-manager  Order "letsencrypt-prod-secret-649678329" completed successfully
  Normal   CertIssued      4s                   cert-manager  Certificate issued successfully

Checking over https on rx name

curl -vkI https://shaungc.com gives:

* Rebuilt URL to: https://shaungc.com/
*   Trying 165.22.171.242...
* TCP_NODELAY set
* Connected to shaungc.com (165.22.171.242) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=shaungc.com
*  start date: Aug 18 21:41:30 2019 GMT
*  expire date: Nov 16 21:41:30 2019 GMT
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fa8e9004200)
> HEAD / HTTP/2
> Host: shaungc.com
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
HTTP/2 200 
< server: openresty/1.15.8.1
server: openresty/1.15.8.1
< date: Sun, 18 Aug 2019 22:43:46 GMT
date: Sun, 18 Aug 2019 22:43:46 GMT
< content-type: text/html
content-type: text/html
< content-length: 1612
content-length: 1612
< vary: Accept-Encoding
vary: Accept-Encoding
< x-amz-id-2: y3bWEJAI5Auw+TFAkSX/xUu9mordIlh1yew7yOf0upm/tkbqfnLH8VhUEjwPCrK4pz/xNU3mwgA=
x-amz-id-2: y3bWEJAI5Auw+TFAkSX/xUu9mordIlh1yew7yOf0upm/tkbqfnLH8VhUEjwPCrK4pz/xNU3mwgA=
< x-amz-request-id: B083269928FF67F2
x-amz-request-id: B083269928FF67F2
< last-modified: Sun, 18 Aug 2019 07:14:59 GMT
last-modified: Sun, 18 Aug 2019 07:14:59 GMT
< etag: "205e390bab2c0cb7ea95f95f39c56178"
etag: "205e390bab2c0cb7ea95f95f39c56178"
< cache-control: max-age=86400
cache-control: max-age=86400
< accept-ranges: bytes
accept-ranges: bytes
< strict-transport-security: max-age=15724800; includeSubDomains
strict-transport-security: max-age=15724800; includeSubDomains

< 
* Connection #0 to host shaungc.com left intact

Only need to look at:

...
* Server certificate:
*  subject: CN=shaungc.com
*  start date: Aug 18 21:41:30 2019 GMT
*  expire date: Nov 16 21:41:30 2019 GMT
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
...

Looks good!

Checking https on subdomains

Run curl -vkI https://aad918e57b0f5cf3544fde08136f9704e71a0b6d.shaungc.com/ gives:

*   Trying 165.22.171.242...
* TCP_NODELAY set
* Connected to aad918e57b0f5cf3544fde08136f9704e71a0b6d.shaungc.com (165.22.171.242) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  start date: Aug 18 22:35:56 2019 GMT
*  expire date: Aug 17 22:35:56 2020 GMT
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fc2df80d800)
> HEAD / HTTP/2
> Host: aad918e57b0f5cf3544fde08136f9704e71a0b6d.shaungc.com
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 404 
HTTP/2 404 
< server: openresty/1.15.8.1
server: openresty/1.15.8.1
< date: Sun, 18 Aug 2019 22:47:03 GMT
date: Sun, 18 Aug 2019 22:47:03 GMT
< content-type: text/plain; charset=utf-8
content-type: text/plain; charset=utf-8
< content-length: 21
content-length: 21
< strict-transport-security: max-age=15724800; includeSubDomains
strict-transport-security: max-age=15724800; includeSubDomains

< 
* Connection #0 to host aad918e57b0f5cf3544fde08136f9704e71a0b6d.shaungc.com left intact

Still not working. This also happens in staging. So always make sure domains are using staging fake certificate first. Our subdomain is not even using cert-manager's certificate, and uses K8's default one.

[rivernews]

Wildcard issue

Seems like it's registering to literally '*.shaungc.com', and is not actually working for subdomains.

Solve by this issue.

---

The wildcard seems to not working at the beginning, when we access subdomain site it's giving nginx backend page. But after awhile it seems working now:

curl -vkI  https://aad918e57b0f5cf3544fde08136f9704e71a0b6d.shaungc.com/

*   Trying 165.22.171.242...
* TCP_NODELAY set
* Connected to aad918e57b0f5cf3544fde08136f9704e71a0b6d.shaungc.com (165.22.171.242) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=shaungc.com
*  start date: Aug 18 21:41:30 2019 GMT
*  expire date: Nov 16 21:41:30 2019 GMT
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fc99680d800)
> HEAD / HTTP/2
> Host: aad918e57b0f5cf3544fde08136f9704e71a0b6d.shaungc.com
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 404 
HTTP/2 404 
< server: openresty/1.15.8.1
server: openresty/1.15.8.1
< date: Mon, 19 Aug 2019 00:40:31 GMT
date: Mon, 19 Aug 2019 00:40:31 GMT
< content-type: text/plain; charset=utf-8
content-type: text/plain; charset=utf-8
< content-length: 21
content-length: 21
< strict-transport-security: max-age=15724800; includeSubDomains
strict-transport-security: max-age=15724800; includeSubDomains

< 
* Connection #0 to host aad918e57b0f5cf3544fde08136f9704e71a0b6d.shaungc.com left intact

To look up the letsencrypt domain registration status, we can always go to https://crt.sh/, and search for the domain name in concern.

[CosmicSage]

Thanks for doing the cray amounts of hardwork, gonna try this solution of yours once my 1 hour ban with production grade certs is lifted, geniusie like you are rare , you should be proud of yourself I know you must be

[rivernews]

I definitely get you for all the stress setting up TLS. Everyone’s config is different but I curated my debug process in this doc, hope it can be useful to you : https://github.com/rivernews/iriversland2-kubernetes/blob/master/docs/progress_tls_cert.md

You might find this issue useful where I integrate another subdomain to use the wilcard certificate, but what I will suggest is to follow this cert-manager official doc to get started. If things go wrong, you can refer to the doc above to help you debug.