TLS server certificate must not contain the CA certificate

[rgl]

Expected Behaviour

The current generated bundle.pem must not contain the CA certificate as that fails the certificate validation.

Only the client must have the CA certificate. The server must not send it.

Current Behaviour

openssl s_client fails to validate the certificate:

root@provisioner:~# echo -e | openssl s_client -showcerts $TINKERBELL_HOST_IP:443 
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 L = @FACILITY@, CN = tinkerbell
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 L = @FACILITY@, CN = tinkerbell
verify return:1
depth=0 L = @FACILITY@, CN = tinkerbell
verify return:1
---
Certificate chain
 0 s:L = @FACILITY@, CN = tinkerbell
   i:L = @FACILITY@, CN = tinkerbell
-----BEGIN CERTIFICATE-----
MIIDpTCCAo2gAwIBAgIUQq5S0pUcxxU4w79dVE0tTljeG5EwDQYJKoZIhvcNAQEL
BQAwKjETMBEGA1UEBwwKQEZBQ0lMSVRZQDETMBEGA1UEAxMKdGlua2VyYmVsbDAe
Fw0yMTA5MTIxMjM4MDBaFw0yMjA5MTIxMjM4MDBaMCoxEzARBgNVBAcMCkBGQUNJ
TElUWUAxEzARBgNVBAMTCnRpbmtlcmJlbGwwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQCynjRTI6Kx37youYrFHpd0hgFGxYkik0DzgCoIRQRFIuxkR6SU
XthNL3tZArogCcqh8jD0MdEcIVlX4mlOVHHaiyEKJd9sxZFvotcSXGhS/upo+/bn
SXha9vtpBFgRyYmyccCXVwNzoDnoRHYL54t3eS+q4AePmHCsua5mthmTF1OJmRrA
rdQ7BvmEMok7Zuk09dUNowdAyDvtIBV7WOLZFtd7uTixebrsQ4L9xcSXZ6zwHp5Q
4TaiigRKz+fmsJI9O1LSNohDUg2tQOQO1VGfMaylYxyl3aIKwSEQXqlIZdUyfsTc
NqZuoOUozF2zMGT7E7oI8t6JvvMIr+AnIC9FAgMBAAGjgcIwgb8wDgYDVR0PAQH/
BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHQYDVR0O
BBYEFDcL3jfy/ZmSI2VMSAsWKnvm3wUPMGsGA1UdEQRkMGKCE3RpbmtlcmJlbGwu
cmVnaXN0cnmCFXRpbmtlcmJlbGwudGlua2VyYmVsbIIKdGlua2VyYmVsbIILdGlu
ay1zZXJ2ZXKCCWxvY2FsaG9zdIcECgMAAocEwKgyBIcEfwAAATANBgkqhkiG9w0B
AQsFAAOCAQEAoOxbT5d3wpuk3DF8d7KA3rvC94US8lO6O9j+omZtIHBtfbSKObVA
2UJYzK5vu7TEFZ776GalNVyZGSqcpgoYEncFCHCwkWFxLa0Ep/U2wh5iZYsFori5
Y8Piwqae0hN0TJTvpHqQlPQ+XTcBg/Y3YguoZfLvBZj9aUxSyWmqp7WYND4oLfXh
wXG7XWdoppHnoqpjeFFZ1iXEVhwEIyKv0vzs7yWZGxrS01V83VMVDNeSFKbasGpx
F0hkA+ZAEzeOKiLQKfIYYK0Lm2dih0+yQzcqgrJ1aCze8jVht9xAxo/aT7xozEDA
5kSWIpr5QRGTWjA1h6+l1tiVpG2r6KlE3Q==
-----END CERTIFICATE-----
 1 s:L = @FACILITY@, CN = tinkerbell
   i:L = @FACILITY@, CN = tinkerbell
-----BEGIN CERTIFICATE-----
MIIDQTCCAimgAwIBAgIUauM0LZhuobLZKZoYIV76UehK2DwwDQYJKoZIhvcNAQEL
BQAwKjETMBEGA1UEBwwKQEZBQ0lMSVRZQDETMBEGA1UEAxMKdGlua2VyYmVsbDAe
Fw0yMTA5MTIxMjM4MDBaFw0yNjA5MTExMjM4MDBaMCoxEzARBgNVBAcMCkBGQUNJ
TElUWUAxEzARBgNVBAMTCnRpbmtlcmJlbGwwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQCoYEtG3Oc726/uYeCSVme6gU6OXvXq2237VJts2AhkpKQJibJm
Hlc+1aaTdSDx7wWsHWj3krKeG9wDdz4QaVUGDTbusKAQh22+odUQd73oL5SOsLSJ
0qRXt8kyvkMPbjkfW2BX9xa+AR2w+6nOtgqMaUYdY8/mYkQuWPRR2+phZAyV/x+V
sEEjWRRJELvfKzIodwSNI9adTOGPHw6yyUguaQyEUxpa33+2AI27bu4cThNspQ1Q
ok3TCi9YwJ2xyNZQ3WTSTCFMsmV03PhBC+30+xwBfYM3ytzjW1rx+aUIKHMvoe94
ISmPu3+MzpD4oSRqqzJJp9WIp76XdF9m1jYLAgMBAAGjXzBdMA4GA1UdDwEB/wQE
AwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRibN7CiS54PIFZiXzXSUv/
7dirHDAbBgNVHREEFDAShwQKAwAChwTAqDIEhwR/AAABMA0GCSqGSIb3DQEBCwUA
A4IBAQBSWhkw324WFjQrd+chAMS5mdyj8wkUQX4gNgkyrVv9QqSzvoZiPGDCRg8X
rHGiKOkZ+ZgjsW2eVsBNctWAO8QTZvTmmXJCUg58Ro+K6d0vGYQVMlsBWXYm2OEg
D+GkmA2wAq41pZJsDH5XP//x1qTDTuAWSYTt5IuKFSN8sbe94D3swhnvQ6SsX5VA
C0RCmfeTqpBqvSSnbUtKFXnuZSDWGwsTCu5/jW0R1pEBAO/XMZeOqHqI+QltrMfZ
0yWuevZKNJwDvXjOTTlXkGwR48dXAfmU9Z6zhnKVkdgFPTcwUnlWDs0YeWeHTETF
qrnwiMjhUj6jc7bTJse0aFLkWKRH
-----END CERTIFICATE-----
---
Server certificate
subject=L = @FACILITY@, CN = tinkerbell

issuer=L = @FACILITY@, CN = tinkerbell

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2350 bytes and written 376 bytes
Verification error: self signed certificate in certificate chain

wget fails to validate the certificate:

root@provisioner:~# wget "https://$TINKERBELL_HOST_IP/v2/_catalog"
--2021-09-12 14:40:59--  https://10.3.0.2/v2/_catalog
Connecting to 10.3.0.2:443... connected.
ERROR: cannot verify 10.3.0.2's certificate, issued by ‘CN=tinkerbell,L=@FACILITY@’:
  Self-signed certificate encountered.
To connect to 10.3.0.2 insecurely, use `--no-check-certificate'.

curl does not fail to validate the certificate, but it should, so I'm not really sure what is going on:

root@provisioner:~# curl "https://$TINKERBELL_HOST_IP/v2/_catalog"
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"registry","Class":"","Name":"catalog","Action":"*"}]}]}

Possible Solution

1. Modify https://github.com/tinkerbell/sandbox/blob/main/deploy/compose/tls/generate.sh#L27 to not bundle the ca into the bundle.pem file. Instead it should split them into a server-crt.pem and a ca.pem file.
2. Modify everything that uses this (at least the following services need to be changed: registry and think-server).
3. When creating the CA certificate, to make the troubleshoot easier, use a different common name for the CA and the server certificate.

Steps to Reproduce (for bugs)

1. Start the provisioner.
2. Execute the command used in the Current Behaviour section above.

[jacobweinstock]

Hey @rgl, thanks for reporting this. I do believe our TLS generation could use some updating. I'm not able to reproduce the curl behavior you're getting. Using the vagrant-virtualbox and vagrant-libvirt setup I'm getting this:

vagrant@ubuntu2004:~$ curl "https://$TINKERBELL_HOST_IP/v2/_catalog"
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I've updated the TLS a bit in this branch here. Would you mind testing out the updated TLS there to see if it has the same behavior?

[rgl]

Sorry, I forgot to describe the root case of the "curl" problem. It was because the update-ca-certificates command refused to add the certificates inside the bundle.pem file to the machine trust store because it has two certificates: the root ca certificate and the server certificate.

Hence, the tls-gen step must be modified to generate two separate certificate files: one for root ca certificate and another for the server certificate. For this you do not need to create an intermediate ca; the existing root ca is enough.

[jacobweinstock]

ah ok. what was the output you got? I was able to add it successfully with just a warning.

vagrant@ubuntu2004:~$ sudo cp -a /vagrant/compose/state/webroot/workflow/ca.pem /usr/share/ca-certificates/cacert.crt
vagrant@ubuntu2004:~$ sudo dpkg-reconfigure ca-certificates
Updating certificates in /etc/ssl/certs...
rehash: warning: skipping cacert.pem,it does not contain exactly one certificate or CRL
rehash: warning: skipping ca.pem,it does not contain exactly one certificate or CRL
1 added, 0 removed; done.
Processing triggers for ca-certificates (20210119~20.04.1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
vagrant@ubuntu2004:~$ curl "https://$TINKERBELL_HOST_IP/v2/_catalog"
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"registry","Class":"","Name":"catalog","Action":"*"}]}]}

[jacobweinstock]

Hey @rgl, any chance you are able to share your output from update-ca-certificates?

[mrmrcoleman]

FYI @rgl @jacobweinstock I can produce the error with curl on vagrant-virtualbox:

$ curl "https://$TINKERBELL_HOST_IP/v2/_catalog"
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

This is currently breaking the instructions in our Vagrant getting started guide and possibly others which I'm working to fix.

If I understand @rgl 's suggestion correctly it seems like the first step would be to fix the gen() function in https://github.com/tinkerbell/sandbox/blob/main/deploy/compose/tls/generate.sh so that instead of bundling server.pem and ca.pem into bundle.pem like this:

cat server.pem ca.pem >"${bundle_destination}"

We instead just put the three separate files into the certs directory like this:

certs
├── server.pem
├── server-key.pem
└── ca.pem

Then I can fix the registry. Currently the registry service in https://github.com/tinkerbell/sandbox/blob/main/deploy/compose/docker-compose.yml looks like this:

 registry:
    image: registry:2.7.1
    restart: unless-stopped
    network_mode: host
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "wget --no-check-certificate https://$TINKERBELL_HOST_IP -O -",
        ]
      interval: 5s
      timeout: 1s
      retries: 5
    environment:
      REGISTRY_AUTH: htpasswd
      REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
      REGISTRY_AUTH_HTPASSWD_PATH: /auth/.htpasswd
      REGISTRY_HTTP_TLS_CERTIFICATE: /certs/${FACILITY:-onprem}/bundle.pem
      REGISTRY_HTTP_TLS_KEY: /certs/${FACILITY:-onprem}/server-key.pem
      REGISTRY_HTTP_ADDR: $TINKERBELL_HOST_IP:443
    volumes:
      - certs:/certs/${FACILITY:-onprem}:ro
      - auth:/auth:rw
    depends_on:
      tls-gen:
        condition: service_completed_successfully
      registry-auth:
        condition: service_completed_successfully

What should the REGISTRY_HTTP_TLS_CERTIFICATE: /certs/${FACILITY:-onprem}/bundle.pem and REGISTRY_HTTP_TLS_KEY: /certs/${FACILITY:-onprem}/server-key.pem be instead?

[mrmrcoleman]

I missed @jacobweinstock 's branch ^^^. Let me work with that and come back.

[rgl]

@mrmrcoleman, yes, that's it, separate in three files: server private key. server certificate. and ca certificate.

I ended doing an workaround in my use of the sandbox:

https://github.com/rgl/tinkerbell-vagrant/blob/master/provision-tinkerbell.sh#L37-L52

[mrmrcoleman]

@rgl I believe by following both your and @jacobweinstock 's instructions I still don't have a working solution, however I've narrowed it down to a couple of things but my TLS foo isn't sufficient. Could we pair on this please?

[rgl]

@mrmrcoleman, can you please verify whether #111 works for you?

please note that I didn't actually test the vagrant environment change. let me known how it goes!