etoews commented 7 years ago

BUG REPORT

Versions

Tectonic version (release or commit hash): 0cf916ee292776ff5ca9bfdc7b5ba391d06a0cec
Terraform version (terraform version): 0.10.7
Platform (aws|azure|openstack|metal): openstack

What happened?

In web browsers, the kube-ca (generated by the tectonic-installer) does not work as a Root CA for certificates that it has signed.

What you expected to happen?

Any certificate signed by the kube-ca should be usable via a web browser as long as the kube-ca has been added to your OS's CA store.

How to reproduce it (as minimally and precisely as possible)?

Create a Kubernetes cluster with the tectonic-installer.

Add the kube-ca to your CA store (this is on a Mac)

sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain generated/tls/ca.crt

Open the apiserver in a web browser (mine is Chrome)

open $(cat $KUBECONFIG | grep server | awk '{print $2}')

Even though the kube-ca has been added as a trusted cert to the system keychain, we still get a NET::ERR_CERT_AUTHORITY_INVALID error.

If you open up the Chrome Developer Tools, go to the Security tab, and View Certificate you see this.

A few issues here:

The kube-apiserver does not show the kube-ca as a parent
The This certificate was signed by an unknown authority error
Numerous <parse error> messages in the Details

Anything else we need to know?

Let's contrast this with what kubeadm does.

Create a Kubernetes cluster with kubeadm on some public cloud

kubeadm init \
    --pod-network-cidr=10.244.0.0/16  \
    --apiserver-advertise-address my.public.ip.addr \
    --apiserver-cert-extra-sans my.public.ip.addr

Download the CA /etc/kubernetes/pki/ca.crt to your local machine and add it to your CA store (this is on a Mac)
```
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ~/Downloads/ca.crt
```
Open the apiserver in a web browser (mine is Chrome)
```
open https://my.public.ip.addr:6443/
```

You get a green security lock.

If you open up the Chrome Developer Tools, go to the Security tab, and View Certificate you see this.

Everything looks good:

The kube-apiserver has kubernetes CA as a parent
This certificate is valid
No <parse error> messages in the Details

Why is this important?

I don't visit the apiserver with a web browser so why do I care about this?

The examples above are just an easy way to illustrate/reproduce the issue.

Our use case is signing certificates with the kube-ca for use with web interfaces (e.g. Grafana) and other systems. It's super convenient to sign certs with kube-ca because:

it's already been generated and is on-hand
it's consistent (and hence more secure I'd argue) to have all of your cluster services signed by the same CA
end users already have the kube-ca in their KUBECONFIG so there no need to distribute a CA to them

etoews commented 7 years ago

Some more debugging info.

The differences I see in the CAs.

kube-ca has a non-zero Serial Number
kube-ca has empty fields in the Issuer and Subject
kube-ca has a X509v3 Subject Key Identifier

kube-ca

$ openssl x509 -text -noout -in generated/tls/ca.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            72:4a:3e:12:a5:e9:78:32:7a:c5:46:19:94:b1:b5:45
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=, ST=, L=/postalCode=, O=bootkube, OU=, CN=kube-ca
        Validity
            Not Before: Oct 17 21:55:43 2017 GMT
            Not After : Oct 17 21:55:43 2018 GMT
        Subject: C=, ST=, L=/postalCode=, O=bootkube, OU=, CN=kube-ca
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    ...
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier:
                63:47:9F:E3:50:97:A7:75:94:66:7B:12:3E:CB:87:B9:4F:C5:D1:48
    Signature Algorithm: sha256WithRSAEncryption
        ...

kubeadm CA

$ openssl x509 -text -noout -in /etc/kubernetes/pki/ca.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)

        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Oct 18 19:43:04 2017 GMT
            Not After : Oct 16 19:43:04 2027 GMT
        Subject: CN=kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    ...
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE

    Signature Algorithm: sha256WithRSAEncryption
        ...

etoews commented 6 years ago

I've confirmed that if I generate the CA certificate/key with kubeadm and use those in the tectonic-installer, that I get a CA cert/key that can sign other certs for use in websites.

./kubeadm alpha phase certs ca --cert-dir /tmp

CA_CERT_ONELINE=$(cat /tmp/ca.crt | tr -d '\n' | sed 's/CERTIFICATE-----/CERTIFICATE-----\\n/g' | sed 's/-----END/\\n-----END/')
echo "tectonic_ca_cert = \"${CA_CERT_ONELINE}\"" >> build/<cluster-name>/terraform.tfvars

CA_KEY_ONELINE=$(cat /tmp/ca.key | tr -d '\n' | sed 's/KEY-----/KEY-----\\n/g' | sed 's/-----END/\\n-----END/')
echo "tectonic_ca_key = \"${CA_KEY_ONELINE}\"" >> build/<cluster-name>/terraform.tfvars

When the cluster is created with those values, I get the same result as I do with a kubeadm created cluster as in the "Anything else we need to know?" section above.

nreisbeck commented 6 years ago

@everett-toews Thanks for getting me pointed in the right direction on this.

Verified bug on AWS & DigitalOcean, when using Self-Signed certs.

The self-signed certificate(s), starting with the CA certificate on down, that are being generated via the TLS Provider are malformed.

RFC 5280 4.1.2.4 for Issuer, and 4.1.2.6 for Subject attributes, all having a lower bound of 1 which appears to be the cause of the parsing errors both in browsers and other certificate consuming services.

RFC 5280 Appendix A provides SIZE(INT..maxSize) constraints for each DN attribute OID.

Refer to X.520 Upper Bounds for DirectoryString{INT:maxSize} maxSize

As detailed by @SpencerBrown in his PR #10 for terraform-provider-tls

modules/tls/kube/self-signed/ca.tf

resource "tls_self_signed_cert" "kube_ca" {
  [...]
  subject {
    common_name  = "kube-ca"
    organization = "bootkube"
  }
  [...]
}

Caused both the Issuer Name and Subject Name to be created as C=, ST=, L=/postalCode=, O=bootkube, OU=, CN=kube-ca.

ca.crt

openssl x509 -text -noout -in build/tectonic/generated/tls/ca.crt 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            4c:c9:75:fc:6b:d0:e4:39:92:8e:2b:4e:8a:26:86:ab
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=, ST=, L=/postalCode=, O=bootkube, OU=, CN=kube-ca
        Validity
            Not Before: Jan 20 17:42:50 2018 GMT
            Not After : Jan 19 17:42:50 2021 GMT
        Subject: C=, ST=, L=/postalCode=, O=bootkube, OU=, CN=kube-ca
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    *snip*
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier: 
                DE:44:14:B1:13:D3:F0:2E:C1:CB:91:A7:DF:A7:06:C9:70:99:F8:AE
    Signature Algorithm: sha256WithRSAEncryption

apiserver.crt

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            50:cb:2d:97:59:68:c4:0c:e7:f9:11:c4:28:a1:e0:e8
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=, ST=, L=/postalCode=, O=bootkube, OU=, CN=kube-ca
        Validity
            Not Before: Jan 20 17:42:53 2018 GMT
            Not After : Jan 19 17:42:53 2021 GMT
        Subject: C=, ST=, L=/postalCode=, O=kube-master, OU=, CN=kube-apiserver
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    *snip*
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier: 
                keyid:DE:44:14:B1:13:D3:F0:2E:C1:CB:91:A7:DF:A7:06:C9:70:99:F8:AE
            X509v3 Subject Alternative Name: 
                DNS:cluster-api.domain.tld, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.3.0.1
    Signature Algorithm: sha256WithRSAEncryption

Using openssl to list the text content of the certificate confirms that there are attributes with empty string values for countryName, stateOrProvinceName, localityName and postalCode were being provided.

CONNECTED(00000003)
depth=1 C = , ST = , L = , postalCode = , O = bootkube, OU = , CN = kube-ca
verify return:1
depth=0 C = , ST = , L = , postalCode = , O = kube-master, OU = , CN = kube-apiserver
verify return:1
---
Certificate chain
 0 s:/C=/ST=/L=/postalCode=/O=kube-master/OU=/CN=kube-apiserver
   i:/C=/ST=/L=/postalCode=/O=bootkube/OU=/CN=kube-ca
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEOjCCAyKgAwIBAgIQUMstl1loxAzn+RHEKKHg6DANBgkqhkiG9w0BAQsFADBc
MQkwBwYDVQQGEwAxCTAHBgNVBAgTADEJMAcGA1UEBxMAMQkwBwYDVQQREwAxETAP
BgNVBAoTCGJvb3RrdWJlMQkwBwYDVQQLEwAxEDAOBgNVBAMTB2t1YmUtY2EwHhcN
MTgwMTIwMTc0MjUzWhcNMjEwMTE5MTc0MjUzWjBmMQkwBwYDVQQGEwAxCTAHBgNV
BAgTADEJMAcGA1UEBxMAMQkwBwYDVQQREwAxFDASBgNVBAoTC2t1YmUtbWFzdGVy
MQkwBwYDVQQLEwAxFzAVBgNVBAMTDmt1YmUtYXBpc2VydmVyMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1QiR54CFPzNCTkPWLSzDa81y0/QNkVcLCN75
uBmyzAB4hdD0C3XsR1RQ675Sa2xKlQVSvG36JhmpSugl1zao+mMSTCHZLbytI+V+
uxcruqlskg0FdSgiXRRibirBGOCAA4m98+SFpmOLA74E3frJwsXYEq3IAaEg2+28
rXLBYAm/hmvpLEd6DAFYOOgOrw/T2iyXHXGMzqjlACaT+jD6GS/fWWmERrpQ2hPu
5vD3ythPwssuaYzoUran0AResy64iNZua817uE/hzN63LphuWd8k68auSJL+8kaZ
IKn7j3Qunvb6nSoTZhZqD6GDFARFzAE3xkKtm+nrG1O9W19p9wIDAQABo4HtMIHq
MA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIw
DAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBTeRBSxE9PwLsHLkaffpwbJcJn4rjCB
iQYDVR0RBIGBMH+CGXRlY3RvbmljLWFwaS5kby53b3Jkcy5meWmCCmt1YmVybmV0
ZXOCEmt1YmVybmV0ZXMuZGVmYXVsdIIWa3ViZXJuZXRlcy5kZWZhdWx0LnN2Y4Ik
a3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FshwQKAwABMA0GCSqG
SIb3DQEBCwUAA4IBAQCflu+lOG6Rtmh2Hw7/GkEn6AN61uk1FBZuCXhnvTMXxq0p
k6nBbMfNHu5sxcCI3KCX76VuaxfVUHRIEKyFVHZE80cf1OH2PuofGwUE34plC3h+
EcJzWPJe/ex++rCY4PDdjj1u/hLPs817sSJKPJPiMgfV+e5vEpxxg1NW6OXOtdtZ
5RLR13Vb7ph4jlTWf+2VkYg3TuZz69wNFhn5IKH5vKlDx85ccvPw+tLX7mlGJO0J
ki3wwOExBF+1wStbm0mt073Ajt52tfCpCiRzgmeTc5J1ZKiF08qvx4jfkIqlBe1b
lL+rCfZdUegNRnWd3YyQYCrvZ3mellbp3QOISJGk
-----END CERTIFICATE-----
subject=/C=/ST=/L=/postalCode=/O=kube-master/OU=/CN=kube-apiserver
issuer=/C=/ST=/L=/postalCode=/O=bootkube/OU=/CN=kube-ca
---
Acceptable client certificate CA names
/C=/ST=/L=/postalCode=/O=bootkube/OU=/CN=kube-ca
Client Certificate Types: RSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA384
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1816 bytes and written 445 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: *snip*
    Session-ID-ctx: 
    Master-Key: *snip*
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket:
    *snip*

    Start Time: 1516839322
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

Empty attributes in the certificate Issuer and Subject dn's were causing other services relying on PKI to fail negotiation.

2018-01-25 05:33:22.503 [debug] <0.193.0> GET https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/default/endpoints/yucky-clownfish-rabbitmq-ha
2018-01-25 05:33:22.505 [debug] <0.212.0> Supervisor inet_gethost_native_sup started undefined at pid <0.213.0>
2018-01-25 05:33:22.505 [debug] <0.60.0> Supervisor kernel_safe_sup started inet_gethost_native:start_link() at pid <0.212.0>
2018-01-25 05:33:22.518 [info] <0.170.0> SSL WARNING: Ignoring a CA cert as it could not be correctly decoded.

2018-01-25 05:33:22.529 [debug] <0.193.0> Response: {error,{failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}},{inet,[inet],{tls_alert,"internal error"}}]}}
2018-01-25 05:33:22.529 [info] <0.214.0> TLS client: In state certify at ssl_handshake.erl:422 generated CLIENT ALERT: Fatal - Internal Error - {unexpected_error,{case_clause,{error,{asn1,{...}}}}}

2018-01-25 05:33:22.529 [debug] <0.193.0> HTTP Error {failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}},{inet,[inet],{tls_alert,"internal error"}}]}
2018-01-25 05:33:22.530 [info] <0.193.0> Failed to get nodes from k8s - {failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}},
                 {inet,[inet],{tls_alert,"internal error"}}]}

Once TLS Provider version was changed to 1.0.1, this provided valid Issuer & Subject names as expected and TLS Handshake(s) were successful.

2018-01-27 03:18:46.974 [debug] <0.198.0> Peer discovery backend initialisation succeeded.
2018-01-27 03:18:46.974 [info] <0.198.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2018-01-27 03:18:46.974 [info] <0.198.0> Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping randomized startup delay.
2018-01-27 03:18:46.974 [info] <0.198.0> K8S API REQUEST STARTING
2018-01-27 03:18:46.974 [info] <0.198.0> /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
2018-01-27 03:18:46.974 [info] <0.198.0> Cert should be above this line
2018-01-27 03:18:46.975 [debug] <0.198.0> GET https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/default/endpoints/busted-woodpecker-rabbitmq-ha
2018-01-27 03:18:47.009 [debug] <0.198.0> Response: {ok,{{"HTTP/1.1",200,"OK"},[{"date","Sat, 27 Jan 2018 03:18:46 GMT"} ...]}
[...]
2018-01-27 03:18:47.009 [info] <0.198.0> k8s endpoint listing returned nodes not yet ready: 10.2.3.59
2018-01-27 03:18:47.009 [info] <0.198.0> All discovered existing cluster peers: rabbit@10.2.2.58
2018-01-27 03:18:47.009 [info] <0.198.0> Peer nodes we can cluster with: rabbit@10.2.2.58
2018-01-27 03:18:47.021 [info] <0.198.0> Node 'rabbit@10.2.2.58' selected for auto-clustering

screen shot 2018-01-26 at 5 17 51 pm

👍

zhex900 commented 6 years ago

Hi, After reading this post, I still don't know how to solve this cert problem. Is there any step by step instructions?

squat commented 6 years ago

@zhex900 let’s discuss in the issue you opened and keep this closed.

coreos / tectonic-installer

The kube-ca does not work as a Root CA for certificates that it has signed #2189