Request Red Hat Certificates for Bare-metal OpenShift Cluster

[rbo]

https://source.redhat.com/groups/public/identity-access-management/identity__access_management_wiki/how_to_submit_a_server_certificate_request_csr_via_command_line

[rbo]

Created key:

$ openssl genrsa 4096 > ocp-coe-cluster.key

CSR:

$ openssl req -config  openssl.ocp-coe-cluster.conf  -sha256 -new -key ocp-coe-cluster.key -out ocp-coe-cluster.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [DE]:
State or Province Name [Bavaria]:
Locality Name [Munich]:
Organization Name [muc.redhat.com]:Red Hat GmbH
Organizational Unit Name [coe]:Munich COE Lab
Common Name [api.cluster.coe.muc.redhat.com]:
Email Address [rbohne+coe@redhat.com]:

openssl req -in ocp-coe-cluster.csr -noout -subject -text | grep -A1 -E '(X509v3|subject)'
                X509v3 Basic Constraints: 
                    CA:FALSE
                X509v3 Key Usage: 
                    Digital Signature, Non Repudiation, Key Encipherment
                X509v3 Subject Alternative Name: 
                    DNS:api.cluster.coe.muc.redhat.com, DNS:*.apps.cluster.coe.muc.redhat.com
--
subject=C = DE, ST = Bavaria, L = Munich, O = Red Hat GmbH, OU = Munich COE Lab, CN = api.cluster.coe.muc.redhat.com, emailAddress = "rbohne+coe@redhat.com"

[rbo]

Your request ID is 9961727

My openssl.ocp-coe-cluster.conf

```ini # OpenSSL root CA configuration file. [ req ] # Options for the `req` tool (`man req`). default_bits = 2048 distinguished_name = req_distinguished_name string_mask = utf8only # SHA-1 is deprecated, so use SHA-2 instead. default_md = sha256 # Extension to add when the -x509 option is used. x509_extensions = v3_ca req_extensions = v3_req [ v3_req ] # Extensions to add to a certificate request basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName = @alt_names [alt_names] DNS.1 = api.cluster.coe.muc.redhat.com DNS.2 = *.apps.cluster.coe.muc.redhat.com # DNS.3 = nginx-ex-ssl-stc-pipeline.6923.rh-us-east-1.openshiftapps.com # DNS.4 = ... # IP.1 = 172.16.0.5 [ req_distinguished_name ] # See . countryName = Country Name (2 letter code) stateOrProvinceName = State or Province Name localityName = Locality Name 0.organizationName = Organization Name organizationalUnitName = Organizational Unit Name commonName = Common Name emailAddress = Email Address # Optionally, specify some defaults. countryName_default = DE stateOrProvinceName_default = Bavaria localityName_default = Munich 0.organizationName_default = muc.redhat.com organizationalUnitName_default = coe emailAddress_default = rbohne+coe@redhat.com commonName_default = api.cluster.coe.muc.redhat.com [ v3_ca ] # Extensions for a typical CA (`man x509v3_config`). subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer basicConstraints = critical, CA:true keyUsage = critical, digitalSignature, cRLSign, keyCertSign [ v3_intermediate_ca ] # Extensions for a typical intermediate CA (`man x509v3_config`). subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer basicConstraints = critical, CA:true, pathlen:0 keyUsage = critical, digitalSignature, cRLSign, keyCertSign [ server_cert ] # Extensions for server certificates (`man x509v3_config`). basicConstraints = CA:FALSE nsCertType = server nsComment = "OpenSSL Generated Server Certificate" subjectKeyIdentifier = hash authorityKeyIdentifier = keyid,issuer:always keyUsage = critical, digitalSignature, keyEncipherment extendedKeyUsage = serverAuth [ crl_ext ] # Extension for CRLs (`man x509v3_config`). authorityKeyIdentifier=keyid:always ```

[rbo]

Received certificate.

$ openssl x509 -in ocp-coe-cluster.crt -noout -dates -subject -issuer 
notBefore=Dec 18 06:34:35 2022 GMT
notAfter=Dec 13 06:34:35 2023 GMT
subject=C = DE, ST = Bavaria, L = Munich, O = Red Hat GmbH, OU = Munich COE Lab, CN = api.cluster.coe.muc.redhat.com, emailAddress = "rbohne+coe@redhat.com"
issuer=O = Red Hat, OU = prod, CN = Certificate Authority

# https://docs.openshift.com/container-platform/4.11/networking/configuring-a-custom-pki.html

$ oc create configmap  user-ca-bundle  -n openshift-config --from-file=ca-bundle.crt=/etc/pki/ca-trust/source/anchors/RH-IT-Root-CA.crt 
configmap/user-ca-bundle created

$ oc patch proxy/cluster --type json -p='[{"op": "replace", "path": "/spec/trustedCA/name", "value": "user-ca-bundle"}]'
proxy.config.openshift.io/cluster patched

# Wait for mcp updated:
$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-cd40f40d8e6787e24e6abc803dec11d2   False     True       False      3              0                   1                     0                      266d
sriov    rendered-sriov-b56214741ab84aed90e25563a2bbaf11    False     True       False      4              1                   1                     0                      14d
worker   rendered-worker-b56214741ab84aed90e25563a2bbaf11   False     True       False      3              0                   0                     0                      266d

[rbo]

Ingress

https://docs.openshift.com/container-platform/4.11/security/certificates/replacing-default-ingress-certificate.html#replacing-default-ingress_replacing-default-ingress

oc create secret tls ocp-coe-cert --cert=ocp-coe-cluster.crt --key=ocp-coe-cluster.key -n openshift-ingress

oc patch ingresscontroller.operator default --type=merge -p '{"spec":{"defaultCertificate": {"name": "ocp-coe-cert"}}}' -n openshift-ingress-operator

API

https://docs.openshift.com/container-platform/4.11/security/certificates/api-server.html#api-server-certificates

$ oc create secret tls  ocp-coe-cert   --cert=ocp-coe-cluster.crt --key=ocp-coe-cluster.key -n openshift-config

$ oc patch apiserver cluster --type=merge -p '{"spec":{"servingCerts": {"namedCertificates": [{"names": ["api.cluster.coe.muc.redhat.com"], "servingCertificate": {"name": "ocp-coe-cert"}}]}}}' 

[rbo]

Problem

$ oc get co/authentication
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication   4.11.13   True        False         True       8m56s   RouterCertsDegraded: secret/v4-0-config-system-router-certs.spec.data[apps.cluster.coe.muc.redhat.com] -n openshift-authentication: certificate could not validate route hostname oauth-openshift.apps.cluster.coe.muc.redhat.com: x509: certificate signed by unknown authority

[rbo]

$ openssl x509 -in /etc/pki/ca-trust/source/anchors/RH-IT-Root-CA.crt -issuer -subject -noout
issuer=C = US, ST = North Carolina, L = Raleigh, O = "Red Hat, Inc.", OU = Red Hat IT, CN = Red Hat IT Root CA, emailAddress = infosec@redhat.com
subject=C = US, ST = North Carolina, L = Raleigh, O = "Red Hat, Inc.", OU = Red Hat IT, CN = Red Hat IT Root CA, emailAddress = infosec@redhat.com
$ openssl x509 -in ocp-coe-cluster.crt -issuer -subject -noout
issuer=O = Red Hat, OU = prod, CN = Certificate Authority
subject=C = DE, ST = Bavaria, L = Munich, O = Red Hat GmbH, OU = Munich COE Lab, CN = api.cluster.coe.muc.redhat.com, emailAddress = "rbohne+coe@redhat.com"
$ openssl verify  -CAfile /etc/pki/ca-trust/source/anchors/RH-IT-Root-CA.crt ocp-coe-cluster.crt 
C = DE, ST = Bavaria, L = Munich, O = Red Hat GmbH, OU = Munich COE Lab, CN = api.cluster.coe.muc.redhat.com, emailAddress = "rbohne+coe@redhat.com"
error 20 at 0 depth lookup: unable to get local issuer certificate
error ocp-coe-cluster.crt: verification failed
$ 

Downloaded the a new propper root ca bundle:

$ openssl verify  -CAfile /tmp/redhat-ca.crt ocp-coe-cluster.crt 
ocp-coe-cluster.crt: OK
$ 

[rbo]

$ oc create configmap redhat-ca --from-file=ca-bundle.crt=/tmp/redhat-ca.crt -n openshift-config
configmap/redhat-ca created
$ oc patch proxy/cluster --type=merge --patch='{"spec":{"trustedCA":{"name":"redhat-ca"}}}'
proxy.config.openshift.io/cluster patched
# Delete old one
$ oc delete configmap/user-ca-bundle -n openshift-config
configmap "user-ca-bundle" deleted

[rbo]

I have still problems with the root ca.

$ oc get configmap -n openshift-authentication-operator trusted-ca-bundle -o yaml | grep -i 'red hat'

Looks like the root ca's are not injected.

[rbo]

OK I did a huge mistake. By default the server should deliver all certificates from Server cert over intermediate until the root CA (w/o root CA) The Client have to know only the root CA not the intermediate.

Here an example:

$openssl crl2pkcs7 -nocrl -certfile <(oc get secrets $SECRET -o go-template='{{ index .data "tls.crt" | base64decode }}' ) | openssl pkcs7 -print_certs  -noout
subject=CN = *.apps.demo.openshift.pub
issuer=C = US, O = Let's Encrypt, CN = R3

subject=C = US, O = Let's Encrypt, CN = R3
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3

$ openssl crl2pkcs7 -nocrl -certfile RH-IT-Root-CA.crt | openssl pkcs7 -print_certs  -noout
subject=C = US, ST = North Carolina, L = Raleigh, O = "Red Hat, Inc.", OU = Red Hat IT, CN = Red Hat IT Root CA, emailAddress = infosec@redhat.com
issuer=C = US, ST = North Carolina, L = Raleigh, O = "Red Hat, Inc.", OU = Red Hat IT, CN = Red Hat IT Root CA, emailAddress = infosec@redhat.com

$ openssl crl2pkcs7 -nocrl -certfile redhat-ca.crt | openssl pkcs7 -print_certs  -noout
subject=O = Red Hat, OU = prod, CN = Certificate Authority
issuer=O = Red Hat, OU = prod, CN = Intermediate Certificate Authority

subject=O = Red Hat, OU = prod, CN = Intermediate Certificate Authority
issuer=C = US, ST = North Carolina, L = Raleigh, O = "Red Hat, Inc.", OU = Red Hat IT, CN = Red Hat IT Root CA, emailAddress = infosec@redhat.com

subject=C = US, ST = North Carolina, L = Raleigh, O = "Red Hat, Inc.", OU = Red Hat IT, CN = Red Hat IT Root CA, emailAddress = infosec@redhat.com
issuer=C = US, ST = North Carolina, L = Raleigh, O = "Red Hat, Inc.", OU = Red Hat IT, CN = Red Hat IT Root CA, emailAddress = infosec@redhat.com

Let's rollout the RH-IT-Root-CA.crt via root ca bundle.

$ oc delete configmap/redhat-ca -n openshift-config
configmap "redhat-ca" deleted
$ oc create configmap redhat-ca --from-file=ca-bundle.crt=RH-IT-Root-CA.crt -n openshift-config
configmap/redhat-ca created
$ 

[rbo]

Created a new chain:

$ openssl crl2pkcs7 -nocrl -certfile ocp-coe-cluster-chain.crt | openssl pkcs7 -print_certs  -noout
subject=C = DE, ST = Bavaria, L = Munich, O = Red Hat GmbH, OU = Munich COE Lab, CN = api.cluster.coe.muc.redhat.com, emailAddress = "rbohne+coe@redhat.com"
issuer=O = Red Hat, OU = prod, CN = Certificate Authority

subject=O = Red Hat, OU = prod, CN = Certificate Authority
issuer=O = Red Hat, OU = prod, CN = Intermediate Certificate Authority

subject=O = Red Hat, OU = prod, CN = Intermediate Certificate Authority
issuer=C = US, ST = North Carolina, L = Raleigh, O = "Red Hat, Inc.", OU = Red Hat IT, CN = Red Hat IT Root CA, emailAddress = infosec@redhat.com

$ 

[rbo]

Apply chain to ingress and api

oc delete secret ocp-coe-cert -n openshift-ingress
oc create secret tls ocp-coe-cert --cert=ocp-coe-cluster-chain.crt --key=ocp-coe-cluster.key -n openshift-ingress

oc delete secret ocp-coe-cert  -n openshift-config
oc create secret tls  ocp-coe-cert    --cert=ocp-coe-cluster-chain.crt --key=ocp-coe-cluster.key  -n openshift-config

[rbo]

Important: at the end of the Cert file and Key, don't forget a newline! Cert & Key will be concatenated for some components and then you have a invalid cert & key file.

And it's strongly recommended: roll out first the root and WAIT until the rollout is done.

[rbo]

Ingress and API RH Cert configurired. Configuration cia GitOps/ArgiCD Secrets with certs direct applied. Certs stored in Bitwarden.