search

cloudfoundry / korifi

Cloud Foundry on Kubernetes
Apache License 2.0
317 stars 65 forks source link

[Feature]: Platform Operator can configure connection between ingress gateway and API Shim to occur over TLS #850

Closed tcdowney closed 2 years ago

tcdowney commented 2 years ago

Blockers/Dependencies

No response

Background

As a Platform Operator I want to be able to configure the connection between our ingress gateway proxy and the API shim to use TLS So that the connection is encrypted and secure

This story will introduce an optional new configuration option for the API Shim: serverCertificateSecretName which, when present, will configure the shim to only accept connections using TLS. When the config is not-present/empty the shim will accept plain HTTP connections (helpful so that we can continue to run it locally during development).

Acceptance Criteria

Is it possible to use kube-rbac-proxy for this?

If we can leverage kube-rbac-proxy as a sidecar container to our API shim that might be convenient. So for local dev we wouldn't run it and we could continue using non-TLS as usual, but in our deployments we would have it and always use TLS. One outstanding question is whether or not you can configure Contour's Envoy to work well with it since we believe it requires client certs that have valid cluster credentials. Anyways, if it works well we can adjust the A/C below to say update our deployment templates and scripts to include it. The outcome we want is one-way TLS to encrypt traffic.

TLS Configured

GIVEN I have created a TLS Secret with a cert for cf-k8s-api-svc.cf-k8s-api-system.service.cluster.local and configured the new serverCertificateSecretName property to point at it WHEN I deploy the API shim THEN I I can see it only accepts connections using TLS and our API Shim HTTPProxy is configured for Upstream TLS AND I confirm I can continue to access the API shim from outside the cluster

TLS Not Configured

GIVEN I have NOT configured the new serverCertificateSecretName property WHEN I deploy the API shim THEN I I can see it accepts plain HTTP connections and its HTTPProxy is not configured for Upstream TLS AND I confirm I can continue to access the API shim from outside the cluster

Dev Notes

tcdowney commented 2 years ago

From @julian-hj during IPM: Should we consider using kube-rbac-proxy for TLS termination?

julian-hj commented 2 years ago

Doing a tiny bit more reading, I think @tcdowney is right that kube-rbac-proxy will require a client token or cert, so probably isn't right for this use case, but another option would be to use an envoy sidecar like this

tcdowney commented 2 years ago

We discussed the Envoy sidecar approach a bit offline -- I think it is nice in general (especially when provided by a service mesh) -- but it will add a lot of overhead (around security and keeping the envoy up to date) and that it's simpler for now to just add the logic for the API shim to be able to "ListenAndServeTLS". Since this is optional, operators are free to bring their own mesh and sidecars to secure this hop.

tcdowney commented 2 years ago

We had a lot of trouble with Contour/Envoy rejecting the cert. I eventually got it working by generating the certs with cert-manager and ensuring that the Issuer field on the certs was non-empty. We could have probably used OpenSSL to set the Issuer field as well, but this just happens to be what I did.

WIP work is on the https://github.com/cloudfoundry/korifi/tree/issues/850 branch.

https://cert-manager.io/docs/configuration/selfsigned/#certificate-validity

tcdowney commented 2 years ago

Documenting my acceptance steps here for posterity:

  1. Inspected the korifi-api-proxy HTTPProxy in korifi-api-system and confirmed it had the expected upstream TLS validation fields set. 
    ...
- name: korifi-api-svc
port: 443
validation:
  caSecret: korifi-api-internal-cert
  subjectName: korifi-api-svc.korifi-api-system.svc.cluster.local
  2. Set up port-forwarding directly to a korifi-api pod to be able to hit it locally 
    kubectl -n korifi-api-system port-forward $(kubectl -n korifi-api-system get pod -l app=korifi-api -o name | head -1) 9002:9000
  3. Used the openssl s_client with the CA for our internal API cert to verify that the connection was using TLS. 
    openssl s_client -connect localhost:9002 -CAfile <(kubectl get secret -n korifi-api-system korifi-api-internal-cert -o jsonpath='{.data.ca\.crt}' | base64 --decode) | openssl x509 -text -noout

Successfully got the server cert out and verified it:

depth=0 O = korifi, CN = korifi-api-svc.korifi-api-system.svc.cluster.local
verify return:1
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            be:12:f5:6e:da:9f:e8:c7:c1:0d:69:b4:dc:e6:02:57
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: O=korifi, CN=korifi-api-svc.korifi-api-system.svc.cluster.local
        Validity
            Not Before: May 11 17:27:18 2022 GMT
            Not After : Aug  9 17:27:18 2022 GMT
        Subject: O=korifi, CN=korifi-api-svc.korifi-api-system.svc.cluster.local
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:c1:ae:8d:c9:85:28:3b:81:1e:dc:91:a2:2e:19:
                    88:72:32:4d:d0:d2:6c:23:be:78:00:65:a1:29:cf:
                    69:61:80:49:52:55:81:9f:91:bf:f7:eb:c0:00:6d:
                    1e:b4:4d:d0:7a:fa:02:46:c6:25:3f:61:9c:cc:79:
                    19:52:ef:7b:e1:74:a7:d7:d0:45:64:60:1e:00:44:
                    2e:c8:c0:0e:11:2c:cb:b0:58:d3:e6:7f:f7:5d:e9:
                    a8:f3:6f:61:06:bb:db:d9:53:85:ce:cb:bc:79:1b:
                    b6:40:a6:93:ca:36:5f:d7:88:4c:94:35:7e:44:51:
                    5c:ea:0f:b5:8e:5d:a9:6b:3b:3c:4a:23:b0:98:16:
                    8a:92:b0:e3:e6:2b:0c:95:27:ac:06:12:36:61:47:
                    e1:66:15:ce:c0:87:11:eb:ad:b6:85:d8:7b:d6:22:
                    87:11:df:40:aa:0a:81:4d:ac:03:42:43:e4:4c:80:
                    a7:0d:9d:57:fc:33:f1:8c:99:c0:69:43:b1:6c:20:
                    b1:4e:2c:11:a4:93:3a:15:d0:37:80:67:d9:7f:a9:
                    ad:24:7f:9e:9d:7e:db:07:a5:23:4c:7c:3b:f1:9e:
                    8e:94:22:4b:15:5a:82:3c:2f:d7:bb:2a:f8:44:60:
                    1c:c0:52:59:55:e8:cc:c3:48:84:2b:20:c1:1f:56:
                    c3:7b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Alternative Name:
                DNS:korifi-api-svc.korifi-api-system.svc, DNS:korifi-api-svc.korifi-api-system.svc.cluster.local
    Signature Algorithm: sha256WithRSAEncryption
         6c:47:cb:94:33:37:dc:ff:56:9f:eb:c8:ce:ff:02:51:47:9f:
         fc:15:57:46:f8:e4:35:78:9e:17:6e:1d:71:e3:2b:e3:91:f7:
         ca:31:bc:e7:f7:92:e5:c3:c1:87:cc:c6:aa:f4:96:7e:26:13:
         0c:21:f1:1a:f0:61:9b:36:1c:02:e4:da:e8:5f:73:5d:49:03:
         3b:78:1a:e6:72:72:96:1b:34:13:d8:de:51:04:38:56:a6:b9:
         c9:45:d8:d8:b4:8f:f7:66:a7:42:ba:d6:2c:c7:a6:67:fb:7f:
         ea:8a:e5:20:d4:eb:3f:c5:e2:8d:13:c8:e6:cd:e2:6f:cb:79:
         67:19:68:97:ac:14:29:08:a3:c8:e6:6a:7c:52:c2:f5:fa:db:
         9f:68:d3:4c:ad:19:7c:80:62:3b:0b:16:4b:b3:88:6b:73:19:
         5f:2b:6d:54:51:8e:04:bf:3e:04:b0:6b:cb:7f:2c:4e:9c:a9:
         92:4e:8c:c2:31:f0:73:7d:26:c9:b8:04:71:2e:b7:0a:13:fb:
         26:a8:0f:f7:8e:8e:ba:01:77:62:08:31:b5:43:17:a4:73:86:
         19:5c:15:d8:0e:2d:af:90:3b:11:e5:9a:b2:21:90:01:2f:e2:
         ca:e0:31:5d:87:1d:2e:26:dc:9c:16:fc:4c:40:69:77:eb:50:
         54:e0:d3:42