can not install: Bootstrap Certs are all expired

[betonmoewe]

Describe the bug

I have tried to install okd with the okd versin of the openshift-installer but without success because of expired certificates:

Version 4.15.0-0.okd-2024-03-10-010116 (latest available version)

How reproducible openshift-install create ignition-configs --log-level debug

Log

WARNING Bootstrap Ignition-Config Certificate aggregator-ca.crt expired at 2024-09-07T09:13:00Z. WARNING Bootstrap Ignition-Config Certificate aggregator-ca-bundle.crt expired at 2024-09-07T09:13:00Z. WARNING Bootstrap Ignition-Config Certificate aggregator-client.crt expired at 2024-09-07T09:13:01Z. WARNING Bootstrap Ignition-Config Certificate aggregator-signer.crt expired at 2024-09-07T09:13:00Z. WARNING Bootstrap Ignition-Config Certificate apiserver-proxy.crt expired at 2024-09-07T09:13:01Z. WARNING Bootstrap Ignition-Config Certificate kube-apiserver-lb-server.crt expired at 2024-09-07T10:58:52Z. WARNING Bootstrap Ignition-Config Certificate kube-apiserver-internal-lb-server.crt expired at 2024-09-07T10:58:53Z. WARNING Bootstrap Ignition-Config Certificate kube-apiserver-localhost-server.crt expired at 2024-09-07T09:13:02Z. WARNING Bootstrap Ignition-Config Certificate kube-apiserver-service-network-server.crt expired at 2024-09-07T10:58:53Z. WARNING Bootstrap Ignition-Config Certificate kube-apiserver-complete-client-ca-bundle.crt expired at 2024-09-07T09:13:03Z. WARNING Bootstrap Ignition-Config Certificate kubelet-client-ca-bundle.crt expired at 2024-09-07T09:13:03Z. WARNING Bootstrap Ignition-Config Certificate kubelet-signer.crt expired at 2024-09-07T09:13:03Z. WARNING Bootstrap Ignition-Config Certificate kubelet-serving-ca-bundle.crt expired at 2024-09-07T09:13:03Z. WARNING Bootstrap Ignition-Config: 13 certificates expired. Installation attempts with the created Ignition-Configs will possibly fail.

[GingerGeek]

This could be caused by reusing an existing installation directory. Did you clear out the install folder or attempt to run the ignition in a totally fresh directory?

[betonmoewe]

yes I have tried this again and again and I have this error when I start the bootstrap server within the server (journalctl -b -f -u release-image.service -u bootkube.service) as well: Sep 10 14:58:02 sfravm-fi-k8s-coreos-okd1-b-46 cluster-bootstrap[3903]: [#1810] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-09-10T14:58:02Z is after 2024-09-07T09:13:02Z

[betonmoewe]

ok .... THANKS for the advice! Very strange ... I have created the install yaml file from scratch in a newly created directory (without any other files (no helper scripts and so on)) ... and ... now I could start the bootstrap server! HURRRAY!! And now I have the next problem :( : when I start the first "master" node the system cloud not download the config from the bootstrap system because of: verify error:num=19:self signed certificate in certificate chain. the cert is

[GingerGeek]

If you re-use an installation directory then by default it will use the certs generated from your first installation in that directory unless you clear it out. You can manually delete the cert files in order to regenerate.

Are both the bootstrap and master nodes generated from the same installation? The cert chain is embedded in the ignitions of both.

[betonmoewe]

ok, I found my error for my first problem ... I only deleted the visible files before each run and overlooked the .openshift_install_state.json (why hidden?)! But I have still no solution for my second problem with missing / wrong certificate for the master installation (ignition merge) ... the .ign for bootstrap and master are from the same openshift-install run. This is the cert from the master.ign:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4819418543862535401 (0x42e203a19adb54e9)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: OU = openshift, CN = root-ca
        Validity
            Not Before: Sep 10 16:20:13 2024 GMT
            Not After : Sep  8 16:20:13 2034 GMT
        Subject: OU = openshift, CN = root-ca
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:cc:ff:96:46:cb:20:da:14:12:48:fe:42:eb:e9:
                    8c:43:2c:ea:0c:35:00:9f:fa:72:e6:74:3f:95:79:
                    33:b9:05:63:96:cc:f8:c7:3d:23:79:65:5d:49:29:
                    b8:d4:bb:e9:6f:69:36:54:a0:20:9e:c5:0d:28:96:
                    10:f7:1a:28:df:d0:b3:f7:cc:0f:2b:72:c9:9e:ef:
                    f6:9b:3d:45:b4:bf:85:07:57:df:76:64:ce:21:ac:
                    b1:0b:aa:dc:a3:1d:89:92:f8:8d:c3:9d:a5:39:ee:
                    66:1a:8b:48:d1:8a:ad:6b:50:5c:d0:75:77:41:63:
                    e5:b7:04:80:a1:e6:34:aa:72:e0:bb:dc:6a:55:56:
                    1b:69:ed:76:ae:9c:48:db:36:bf:2a:37:31:c2:77:
                    c9:0d:18:12:b5:01:9f:22:51:01:1a:20:d5:67:91:
                    41:bd:62:d3:96:4e:06:9b:77:b3:fc:16:d6:d5:f2:
                    7e:5d:65:f0:58:65:58:77:c8:e9:ea:68:53:23:57:
                    d7:f3:aa:c4:65:a3:b1:a7:f3:f5:a5:6f:19:24:ca:
                    d1:10:11:df:a4:bb:f7:91:4f:7f:c5:7a:07:7d:af:
                    9a:67:ca:98:6a:c3:ca:e7:9a:b7:cf:5a:88:bc:c9:
                    f3:91:11:2e:c4:c3:2a:c5:fa:a2:15:58:96:0a:a3:
                    fc:6b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier: 
                78:63:A5:16:82:DC:E8:C2:B9:D0:DD:9A:14:2C:88:E6:10:B2:2F:EB
    Signature Algorithm: sha256WithRSAEncryption
         96:a9:3d:4a:67:d4:af:bf:00:0f:68:e5:95:67:ea:de:dc:96:
         94:b7:7d:69:7c:d8:66:3a:f4:06:0d:a3:67:2e:7a:5b:fb:2c:
         99:fe:b8:9c:2c:0a:2f:6f:9f:52:ca:c6:d2:6e:65:11:c6:d9:
         f9:c8:e7:6b:60:b9:f8:b8:5a:77:d6:45:1c:20:8f:ab:2d:68:
         35:71:1c:1e:80:ec:46:1b:a7:aa:86:05:5d:d2:d1:34:0f:9d:
         cc:49:02:c7:0f:12:06:58:ae:cc:a6:63:74:9f:e3:d2:ac:b6:
         df:ce:c3:aa:9c:00:03:53:3a:75:7c:c7:1a:0b:56:83:af:9a:
         1e:89:d0:07:af:3c:2c:ca:bc:32:29:32:c1:ca:03:48:75:45:
         5b:5f:0f:a9:f7:3f:05:50:45:97:4e:c8:70:9c:d1:0c:6d:46:
         81:18:2b:1f:77:74:77:15:b7:b7:bb:24:82:9e:b3:de:22:0a:
         ac:d2:6f:a0:e3:b2:63:f6:16:54:d6:db:64:12:c0:8f:b8:a6:
         76:f1:e4:b6:a1:d3:57:21:fe:db:1a:ca:39:6a:b7:3c:7a:63:
         24:a4:24:1a:09:a9:41:ca:e2:bd:9c:ed:06:42:c2:dc:59:31:
         22:8c:24:a3:c2:cd:72:5d:06:20:70:e4:fc:b0:c3:4f:a6:b5:
         e5:49:a9:61

and this is the output from openssl s_client --showcerts --connect api-int.okd1.fi-acp.priv:22623 </dev/null:

CONNECTED(00000004)
depth=1 OU = openshift, CN = kube-apiserver-lb-signer
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 OU = openshift, CN = kube-apiserver-lb-signer
verify return:1
depth=0 O = kube-master, CN = system:kube-apiserver
verify return:1
---
Certificate chain
 0 s:O = kube-master, CN = system:kube-apiserver
   i:OU = openshift, CN = kube-apiserver-lb-signer
-----BEGIN CERTIFICATE-----
MIIDizCCAnOgAwIBAgIIYwuWdQGk45EwDQYJKoZIhvcNAQELBQAwNzESMBAGA1UE
CxMJb3BlbnNoaWZ0MSEwHwYDVQQDExhrdWJlLWFwaXNlcnZlci1sYi1zaWduZXIw
HhcNMjQwOTExMDkxMDQ5WhcNMjQwOTEyMDkxMDUyWjA2MRQwEgYDVQQKEwtrdWJl
LW1hc3RlcjEeMBwGA1UEAxMVc3lzdGVtOmt1YmUtYXBpc2VydmVyMIIBIjANBgkq
hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwWY2GLspWwRjdJGTARRYoPAqvcGUefKA
gSNgHZUmWJJ+9zU5bC90qqRHivt+DghlyVo1ps1zlXFduDV13o5Tzh7K6Y9AB/tp
UaCgya/ULqkRfhIIPOpHTPw0unD037zXojUkul/5qUYbxVV0qLdjtL7KSZPMa7Ti
AdLijSs7BTLx5NOv0F5iNhu3fE+Bko/MZUksyjdTnzbhSUpkqKetPNApvlopWwoZ
I5pbUnTLV8Kqpwk02f2NYrskBR/CLmxsNcijlS3VI/E2ED6sAhFTeQp3vpb4a6lZ
9Ugse2Wp99dmCafgI+iiFmBhe4lXyg/sRkpXdiZj8JxuDu+PB59GnwIDAQABo4Gb
MIGYMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMB
Af8EAjAAMB0GA1UdDgQWBBSxjawuNLaPb+MH7DXRxjqTVO6nHTAfBgNVHSMEGDAW
gBQPP+inNJRr1bMjOaFHod6kdPI2GzAjBgNVHREEHDAaghhhcGktaW50Lm9rZDEu
ZmktYWNwLnByaXYwDQYJKoZIhvcNAQELBQADggEBAEa91TJ1r1+juApYkMRotoZT
lrLtrNEYOKfk3RhKsrGMnlcdPoef1LMMFWqctRPqtcXMzYn3u5N9GvdSVVIm16tL
Xunbn2HO7Ct8H1LacNhIFMJ3OhGwGLgQ0JLL0OY1rTlKOdDUFDehHbyCHDi+DIH3
Qg9ldvPM8zlr83zyw8wil89r3zJzqTdmhzddmdSLjnYZi+grt6UItErddcIPsNDk
EGDx5gbczSMXjDoNgbZ9lsI2S68qKnsM042/7uZmN6k9o9ntUWHtvzfGzOZYKOss
0IWqf2X50buEnT7ieXzyeZS1mE4wE6Kj0qe/KRf+nfCahIneprcHvt4ONEm9kos=
-----END CERTIFICATE-----
 1 s:OU = openshift, CN = kube-apiserver-lb-signer
   i:OU = openshift, CN = kube-apiserver-lb-signer
-----BEGIN CERTIFICATE-----
MIIDMjCCAhqgAwIBAgIIPjVB0FMuCpEwDQYJKoZIhvcNAQELBQAwNzESMBAGA1UE
CxMJb3BlbnNoaWZ0MSEwHwYDVQQDExhrdWJlLWFwaXNlcnZlci1sYi1zaWduZXIw
HhcNMjQwOTExMDkxMDQ5WhcNMzQwOTA5MDkxMDQ5WjA3MRIwEAYDVQQLEwlvcGVu
c2hpZnQxITAfBgNVBAMTGGt1YmUtYXBpc2VydmVyLWxiLXNpZ25lcjCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBAKp6eBh/4zH8WPDYyZf8DT1hZhzqOze+
Nsad6etJY2llcdioyAJafa1dQJVq8t5sD7F7/uKASh+pU8U4ZSmo8x4CuE6FpATJ
J0GSXmP6A+hTPnXXomVy1mhYhWrGi91p68tItgFFkx1OwK5iJCXzyg4SSG9C8BO+
fvgh86XNH5wqKTFx8vDVZEXTUxzVDpadJQ4cOj2xZzz/X/+2jenws3P0/J1g7OsK
zIBxBY1leIqwZiBm/Qv+t1VOPKo1Y8C/EFc7YYCGIcBP2v3/KHaMyDIptaZflFQJ
tb/1zahEnl1dR1ZS97gj5/TBmpjgWVetPd5x7kvZSdFWRSEHhdKupeUCAwEAAaNC
MEAwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFA8/
6Kc0lGvVsyM5oUeh3qR08jYbMA0GCSqGSIb3DQEBCwUAA4IBAQBNZlzrqNmFdrPf
LdzFmHVbTp5SvU2BycGwanoILLHu2g6tIzSkF/sfHEVifNedC7VbzwNP1ZVQk0Lf
Qyy4ltxs//j9MUVoMNkVV7QZPTUF8Zj3mfr+yjmZmyci0xSD9XMP2ei0d5lO96Nk
+YAm4kWfoQ8eHb3579HH/T6bA9/lqzmPH95dT5p1R/w6mBZ/zRnW/ojEp6Z+Wl/q
qD/rVFkAubgyZll8KrvmBbjijHtzh+BYphfFdYmzR3YgDhTLqKTMabBYvkWoNHyp
FWc36oClVxgZidcSD7CmiAR0xs5t/AaOD6G8MhyTuAJHvyksbPjxQFa/3WY4mSts
0v/28/3V
-----END CERTIFICATE-----
---
Server certificate
subject=O = kube-master, CN = system:kube-apiserver

issuer=OU = openshift, CN = kube-apiserver-lb-signer

---
Acceptable client certificate CA names
OU = openshift, CN = admin-kubeconfig-signer
OU = openshift, CN = kubelet-signer
OU = openshift, CN = kube-control-plane-signer
OU = openshift, CN = kube-apiserver-to-kubelet-signer
OU = openshift, CN = kubelet-bootstrap-kubeconfig-signer
OU = openshift, CN = aggregator-signer
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2707 bytes and written 416 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self signed certificate in certificate chain)
---
DONE

is this ok?

[betonmoewe]

ok ... failure found: cut&paste error in haproxy.conf :( wrong backend port for 22623 (instead of 22623 6443) ... thanks for the help ... I close