Error "Unable to verify first certificate" on connection

[omar-nahhas]

I cannot log-in into a cluster if my cluster ca is in the form of certificate-authority-data

(from .kube/config ...)

- cluster:
    certificate-authority-data: asdfasdfasdfasdfasfdasdfasdfasdf

[astefanutti]

I've just tried connecting with a cluster certificate-authority-data configured, but haven't been able to reproduce.

It should be supported: https://github.com/astefanutti/kubebox/blob/68fd5335626fb3b643da032cd8de7877e8750970/lib/config/context.js#L67-L69

Could you confirm that the certificate-authority-data field is base64 encoded. Besides, could you provide more information on your environment so that we can try reproducing?

[loffelmacher]

Getting the same error message here. Using a username/password in the login box, I am able to connect to the cluster using kubectl. I provided an URL to an AWS ELB for the URL in login box, same one I found in my kube config.

[astefanutti]

It seems the "Unable to verify first certificate" is returned when some intermediate certificates aren't bundled along with the server certificate. It may actually work with kubectl because it's more permissive than Node.

[astefanutti]

@loffelmacher Would you be able to share more info about your kube config content and the server certificate that may help identifying the root cause?

[infinitydon]

Using kubeconfig file like below (just an example) is failing with Authentication failed "Unable to verify first certifcate"

apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://api.tesususud
  name: tesususu
contexts:
- context:
    cluster: tesususu
    namespace: kube-system
    user: admin
  name: tesususu
current-context: tesususu
kind: Config
preferences: {}
users:
- name: admin
  user:
    password: QAjtesususu
    username: admin

I think the line "insecure-skip-tls-verify: true" enables kubectl to skip the certificate verification

[sabrehagen]

I'm experiencing the same issue. Here's my kubeconfig:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURDe...redacted...=
    server: https://104.xxx.xxx.xxx
  name: staging
contexts:
- context:
    cluster: staging
    user: staging
  name: staging
kind: Config
preferences: {}
users:
- name: staging
  user:
    auth-provider:
      config:
        access-token: ya29.redacted
        expiry: 2018-10-21T12:10:30.613021286Z
      name: gcp

[johnpoth]

Hi @sabrehagen,

Looks like you are using the auth-provider section, which is slightly different. We currently don't read that section. See #14

Task is tracked by https://github.com/astefanutti/kubebox/projects/1#card-10901439

Thanks!

[johnpoth]

@infinitydon the 'insecure-skip-tls-verify' flag is taken into account in kubebox (I use it all the time when running clusters locally). Are you experiencing the issue in the browser or when using node?

[infinitydon]

@johnpoth -- I am facing this problem any time I initiate kubebox from my bash shell. And that is the format of the kubeconfig I am using, I can communicate with the cluster using kubectl without any issues.

Is there some way to capture some log or debug?

[astefanutti]

One possible cause could be the difference between root CA sourced from Node compared to Golang.

Golang sources from: https://golang.org/src/crypto/x509/root_linux.go While Node sources from: https://github.com/nodejs/node/blob/v11.x/src/node_root_certs.h

The NODE_EXTRA_CA_CERTS environment variable is used by Node as a way to add extra CA certificates, as documented in https://nodejs.org/api/cli.html#cli_node_extra_ca_certs_file. It must be a file containing certificates in PEM format.

From the Node documentation, neither the well known nor extra certificates are used when the ca options property is explicitly specified for a TLS or HTTPS client, which is what Kubebox is doing when the certificate-authority-data field is present in the kubeconfig file. So that may explain why the error occurs when certificate-authority-data is present.

It'd be awesome if someone facing the issue could try setting the NODE_EXTRA_CA_CERTS, with a file containing certificates from https://golang.org/src/crypto/x509/root_linux.go.

For the error occurring with certificate-authority-data present in kubeconfig file, it may be that intermediate certificates have to be bundled in the field, e.g.:

$ cat \
 cert.pem \
 intermediate-cert.pem \
 ...
 > fullchain.pem

If someone facing the issue for that case could be doing the test, that'd be awesome as well.

Otherwise, it'd be great if someone facing the issue could confirm the above, by providing the server certificate full chain, and the non-redacted certificate-authority-data value if present.

[cscetbon]

@astefanutti I created a file with all my certificates in it and used NODE_EXTRA_CA_CERTS to reference that file. Not much luck ...

$ k version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.0", GitCommit:"0ed33881dc4355495f623c6f22e7dd0b7632b7c0", GitTreeState:"clean", BuildDate:"2018-09-28T15:20:58Z", GoVersion:"go1.11", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.4+coreos.0", GitCommit:"5e73e0769e5c4ac497235e2817868b1a37032fba", GitTreeState:"clean", BuildDate:"2018-03-12T20:05:58Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
$ node --version
v10.12.0
$ cat ~/.kube/creds/dev2/*.pem > ~/Downloads/ca-files.pem
$ NODE_EXTRA_CA_CERTS=~/Downloads/ca-files.pem kubebox

But I'm using certificate-authority in my kube config with client-certificate and client-key, not certificate-authority-data so that might be not relevant to your tests ...

[astefanutti]

@cscetbon Thanks! Here are my requests:

• Could you provide your kubeconfig file relevant info?
• What Kubernetes setup do you use?
• Could you provide more info about your server certificate, i.e., its certificate full chain?
• In the NODE_EXTRA_CA_CERTS, could you add the certificates from https://golang.org/src/crypto/x509/root_linux.go?
• If you're using certificate-authority or certificate-authority-data in your kubeconfig file, could you add the intermediate certificates if any, with the least authoritative certificate first - your server's certificate, followed by the furthest missing intermediate, and then the next closest to the root, etc (without the root one).

[cscetbon]

Could you provide your kubeconfig file relevant info?

apiVersion: v1
clusters:
- cluster:
    certificate-authority: creds/dev2/ca.pem
    server: https://master.dev2.test.xx.com
  name: dev2
- context:
    cluster: dev2
    namespace: my_ns
    user: dev2-cscetbon
  name: dev2
- name: dev2-cscetbon
  user:
    client-certificate: creds/dev2/cscetbon.pem
    client-key: creds/dev2/cscetbon-key.pem

Could you provide more info about your server certificate, i.e., its certificate full chain?

I don't have this information

What Kubernetes setup do you use?

wdym ? 1.9.4 nodes running coreOS

If you're using certificate-authority or certificate-authority-data in your kubeconfig file, could you add the intermediate certificates if any, with the least authoritative certificate first - your server's certificate, followed by the furthest missing intermediate, and then the next closest to the root, etc (without the root one).

As you can see I user certificate-authority but I don't have access to any intermediate certificates and don't know if there are

[astefanutti]

@cscetbon, thanks.

Could you provide more info about your server certificate, i.e., its certificate full chain?

I don't have this information

You can run the following command:

$ openssl s_client -servername master.dev2.test.xx.com -connect master.dev2.test.xx.com:443 </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' >/path/to/certificate.pem

And provide the content for creds/dev2/ca.pem, or check wether it contains all the certificates chain listed above.

[cscetbon]

I ran your command and got one certificate back. I got also a few errors back like

...
verify error:num=20:unable to get local issuer certificate
...
verify error:num=27:certificate not trusted
..
verify error:num=21:unable to verify the first certificate
verify return:1
poll error

I compared the returned certificate to ~/.kube/creds/dev2/ca.pem and it's different. I then tried to add that one to ~/Downloads/ca-files.pem and connect using kubebox but still got the same issue.

[astefanutti]

@cscetbon thanks. Interesting, it seems OpenSSL is complaining about the server certificate too, which is likely why Node fails as well.

Would you be able to provide the certificate chain printed by the following command (I don't think there is any sensitive information there), e.g.:

$ openssl s_client -connect google.com:443
CONNECTED(00000003)
depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = US, O = Google Trust Services, CN = Google Internet Authority G3
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google LLC/CN=*.google.com
   i:/C=US/O=Google Trust Services/CN=Google Internet Authority G3
 1 s:/C=US/O=Google Trust Services/CN=Google Internet Authority G3
   i:/OU=GlobalSign Root CA - R2/O=GlobalSign/CN=GlobalSign
---
...

It seems some certificates also include an "Authority Information Access" (AIA) field with intermediate CA Issuers. And intermediate certificates are not automatically fetched by Node.js (nodejs/node#16336), nor Firefox (https://bugzilla.mozilla.org/show_bug.cgi?id=399324), while Chrome implements AIA. I wonder whether Golang supports AIA or it looks for extra CA elsewhere.

You could try with -CApath /etc/ssl/certs/ to find out, e.g.:

$ openssl s_client -CApath /etc/ssl/certs/ -connect master.dev2.test.xx.com:443

Last but not least, you can check for the CA Issuers field in the server certificate with:

$ echo -n | openssl s_client -servername master.dev2.test.xx.com -connect master.dev2.test.xx.com:443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > master.dev2.test.xx.com.crt
$ openssl x509 -in master.dev2.test.xx.com.crt -text -noout

Some references:

• https://stackoverflow.com/questions/7587851/openssl-unable-to-verify-the-first-certificate-for-experian-url
• https://github.com/nodejs/node/issues/16336
• https://github.com/openssl/openssl/issues/5168

[cscetbon]

Here is all that I can get from the different commands :

---
Certificate chain
 0 s:/CN=kube-apiserver
   i:/CN=kube-ca
---
Server certificate
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
subject=/CN=kube-apiserver
issuer=/CN=kube-ca
---
Acceptable client certificate CA names
/CN=kube-ca
---
SSL handshake has read 1622 bytes and written 456 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 784C5453D79A9F247491CC22DD1B911F47FDC4960461AFCD79C1835977A28CC7
    Session-ID-ctx:
    Master-Key:  ...
    TLS session ticket: ...

    Start Time: 1540296680
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)

$ echo -n | openssl s_client -servername master.dev2.test.xx.com -connect master.dev2.test.xx.com:443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > master.dev2.test.xx.com.crt
$ openssl x509 -in master.dev2.test.xx.com.crt -text -noout 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 12952900851096730370 (0xb3c1f2b25879bb02)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kube-ca
        Validity
            Not Before: Mar  9 21:29:26 2018 GMT
            Not After : Mar  9 21:29:26 2019 GMT
        Subject: CN=kube-apiserver
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    ....
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Key Usage:
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Subject Alternative Name:
                DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:master.dev2.test.xx.com, DNS:master0.dev2.test.xx.com, DNS:master1.dev2.test.xx.com, DNS:master2.dev2.test.xx.com, DNS:localhost, IP Address:172...., IP Address:172...., IP Address:172..., IP Address:172...
    Signature Algorithm: sha256WithRSAEncryption
         .....

[astefanutti]

@cscetbon thanks. It doesn't look like there are any intermediate certificates.

Could you try running the following:

$ openssl s_client -CAfile creds/dev2/ca.pem -connect master.dev2.test.xx.com:443

And provide the content of creds/dev2/ca.pem?

[cscetbon]

@astefanutti I got more information. So I talked to someone who has all the permissions and is using the same way as me to connect. Kubebox works well for him ! The way we do is as follow :

• The CA and CA-key live in vault
• We issue a request to Vault (Hashicorp) to generate a certificate and key based on our names from the CA
• We write them on disk (the pem files you see in my config)

So for some reason, my certificates do not allow me to use kubebox, and it seems to be because of a permission somehow, or a local library but not because of the certificates as they are generated the same way

[cscetbon]

In the README I see Currently requires priviledged access / role. Could it be because of it ? But the error I get says "Authentication failed"

[astefanutti]

@cscetbon

In the README I see Currently requires priviledged access / role. Could it be because of it ? But the error I get says "Authentication failed"

This note only applies to fetch resource usage data. So the rest should work fine as well as the connection to the API server, given kubectl works.

[cscetbon]

@astefanutti Then there is something else ...

[astefanutti]

@astefanutti Then there is something else ...

@cscetbon yes, there must be. By chance would you be able run kubectl get --raw /, which is the same endpoint Kubebox is first checking before asking for authentication if that return 401 or 403.

[cscetbon]

$ kubectl get --raw /
Error from server (Forbidden): forbidden: User "cscetbon" cannot get path "/"

[astefanutti]

@cscetbon let me re-open #21 then.

[astefanutti]

As a work-around, it seems adding the certificate using the NODE_EXTRA_CA_CERTS environment variable works, e.g.:

$ NODE_EXTRA_CA_CERTS=CA.crt kubebox

where CA.crt is the Kube config certificate-authority path or the certificate-authority-data base 64 decoded (echo <certificate-authority-data> | base64 -D > CA.crt).

I still need to understand why passing the CA certificate to the HTTPS request doesn't work.

[astefanutti]

It should be fixed with 792c0c8b6b9e6b65afd52a6117b54ea2b01bf49c. The root cause was that CA option was overriding the whole list of CAs instead of appending to the list of existing root CAs.