search

cloudfoundry-incubator / kubo-deployment

Contains manifests used to deploy Cloud Foundry Container Runtime
https://www.cloudfoundry.org/container-runtime/
Apache License 2.0
275 stars 114 forks source link

Wrong client ca certificate configuration #374

Closed hanlins closed 5 years ago

hanlins commented 5 years ago

What happened: In /var/vcap/jobs/kube-apiserver/config/bpm.yml we have - "--client-ca-file=/var/vcap/jobs/kube-apiserver/config/kubernetes.pem", but kubernetes.pem is not even a CA.

master/3aee2664-bb73-4456-8d24-cb361ce19ecf:/var/vcap/jobs/kube-apiserver/config# openssl x509 -in kubernetes.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            32:c0:36:db:10:61:10:3c:22:f7:8c:af:34:01:93:19:c7:3c:ae:e9
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=ca
        Validity
            Not Before: Jan 31 04:51:44 2019 GMT
            Not After : Jan 31 04:51:44 2023 GMT
        Subject: CN=b7c37984-56f2-4285-aa84-d8ca745a9684.internal, O=system:masters
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:aa:35:a0:1a:9e:2e:6f:a3:ee:ca:1e:19:2b:bc:
                    18:2f:17:ae:44:39:b3:fd:94:2d:07:dc:47:6a:b9:
                    9e:77:77:61:8e:6d:cb:61:94:91:5a:e7:1e:ca:ba:
                    e5:29:7f:3d:58:1f:a9:d4:21:57:3e:02:b3:5e:4c:
                    a5:e7:aa:8e:a5:a0:7f:fe:52:71:4d:49:ba:30:c9:
                    55:a0:05:fc:4f:63:29:22:ba:48:9c:ac:60:4c:6c:
                    cf:7b:c1:37:89:0c:25:17:cf:0f:5f:f7:b1:30:19:
                    41:da:1c:f7:cb:60:69:78:78:6c:37:86:c2:ec:96:
                    57:fb:0d:df:c2:89:7d:29:40:d0:97:3d:3b:0a:47:
                    4d:e9:0e:48:1e:cb:cb:20:2a:41:67:0c:08:94:b2:
                    f1:10:57:0c:a6:90:cd:5b:69:d7:f0:f9:de:41:2c:
                    61:7c:fe:a7:7e:33:dc:42:85:21:2a:4b:91:34:9a:
                    54:f8:9c:cf:60:bc:bc:67:54:e6:a8:a6:d8:fd:51:
                    2d:75:e4:c8:ac:46:11:d5:fa:25:58:3c:8c:d3:e3:
                    25:fa:9a:85:f1:70:8c:cc:06:bb:d7:2e:d1:63:41:
                    a5:34:7e:72:95:65:19:a8:8a:20:e2:89:17:20:6d:
                    86:91:c6:68:b3:f0:e8:dd:f0:25:8e:d9:16:5a:ba:
                    d8:91
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                E7:35:1C:C1:52:CD:0A:EE:EC:79:13:EB:61:59:D4:E9:A5:F7:AC:4D
            X509v3 Subject Alternative Name:
                IP Address:10.100.200.1, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:master.cfcr.internal, DNS:external, DNS:b7c37984-56f2-4285-aa84-d8ca745a9684.internal
            X509v3 Authority Key Identifier:
                keyid:9A:0A:6B:58:4E:76:6B:8E:29:03:80:8D:4C:59:19:BC:DC:79:22:A1

            X509v3 Basic Constraints: critical
                CA:FALSE
    Signature Algorithm: sha256WithRSAEncryption
         94:5b:70:06:0f:53:78:09:8f:85:7e:f3:7c:96:91:b5:20:ae:
         3d:c3:db:ae:ad:4b:a3:28:93:88:81:39:a3:2b:94:5a:85:f0:
         dc:ae:47:2d:1f:38:98:da:82:5d:50:72:d3:cc:3b:52:52:89:
         c8:1a:d6:96:a8:02:0e:8a:fc:70:3a:21:5b:67:46:a5:c3:e0:
         e5:9c:6e:64:0f:65:5c:69:a2:1c:9c:b0:db:99:c6:31:b2:87:
         d8:41:92:8c:b7:55:8a:9c:e3:7e:f5:64:70:fb:ab:44:dd:bb:
         5a:77:5f:b3:bb:d6:57:d2:f8:ea:a7:35:3b:27:e0:12:dc:ba:
         72:1e:c5:dd:2b:b7:14:b2:43:09:89:36:04:29:21:b0:14:62:
         95:e3:e3:da:3e:5b:af:f0:0a:f0:64:75:26:3f:95:cc:c0:78:
         7f:98:c1:51:3e:ec:05:c2:17:e6:53:bc:7d:b4:b8:13:76:c4:
         93:a6:12:5a:57:8c:36:3e:90:0b:10:7f:f5:5a:7d:bf:8b:e1:
         82:b0:ad:7d:a6:5a:7f:a0:14:2f:90:f2:e7:67:a3:43:c8:85:
         0d:62:d0:d2:84:e7:cc:82:27:a5:8d:0d:9d:ec:a0:d4:43:9a:
         ed:f0:2a:06:2f:5d:b4:81:0b:db:1e:4a:8b:33:3f:d8:83:a2:
         2c:de:da:49

What you expected to happen: Instead of using kubernetes.pem, we should use kubernetes-ca.pem instead. Manually tried and works.

How to reproduce it (as minimally and precisely as possible): This is PKS development related, simply deploy the latest dev-tile (matched with p-pks-integration's master branch) with nsx-t, it will fail when creating cluster, with job nsx_kube_proxy.

Anything else we need to know?: Just fix this line: https://github.com/cloudfoundry-incubator/kubo-deployment/blob/master/manifests/cfcr.yml#L215, change it from kubernetes.pem to kubernetes-ca.pem

Environment:

tvs commented 5 years ago

Hi @hanlins,

It looks like you're trying to integrate the CFCR version of kubo-deployment with the PKS fork of kubo-release. As of this moment, that file (kubernetes-ca.pem) does not exist in kubo-release: https://github.com/cloudfoundry-incubator/kubo-release/tree/develop/jobs/kube-apiserver/templates/config

This fix was already made in the PKS fork of kubo-deployment here: https://github.com/pivotal-cf/pks-kubo-deployment/commit/4ad7e9c68dbe4880433a744ad96f9392022325b4

I'm in the process of bringing both of those changes into upstream CFCR as they help facilitate certificate rotation, but you should be careful about mixing the manifest and release from CFCR and the PKS fork.

hanlins commented 5 years ago

Got it, thanks for the info.