"failed to find PEM block" after successfully configuring ingress controller with custom certs on OCP 4.6.4

[reggie-k]

I have a custom wildcard cert, which I managed to configure IngressController with, after much struggle.

I chained the cert, intermediate cert, root cert and the key into the pem, and only then ingress controller was up and I finally saw that indeed my custom cert is served (for example upon accessing the console route).

However, now the oauth-openshift complains that:

Copying system trust bundle I0203 14:17:38.623709 1 dynamic_serving_content.go:111] Loaded a new cert/key pair for "serving-cert::/var/config/system/secrets/v4-0-config-system-serving-cert/tls.crt::/var/config/system/secrets/v4-0-config-system-serving-cert/tls.key" F0203 14:17:38.624117 1 cmd.go:49] failed to load SNI cert and key: tls: failed to find PEM block with type ending in "PRIVATE KEY" in key input after skipping PEM blocks of the following types: [CERTIFICATE CERTIFICATE CERTIFICATE] g

I take it that the ingress controller operator, upon creation of my custom cert secret, also updates the v4-0-config-system-serving-cert secret that the openshift-oauth uses..?

How should the pem be constructed so that both the ingress controller and the oauth pod are at peace?

[reggie-k]

Works just fine without the key in the pem, only the cert + intermediate + root. The problem was the ansible filter used to base64 encode the secret - sane secret was encoded differently by OCP and ansible. Adding an ansible filter made the secret be created on the format OCP had expected.

[stlaz]

Thank you for sharing

[nate-duke]

@reggie-k could you elaborate on what the issue was here? We've just renewed our wildcards and are now seeing this same thing. That being that the router is happy with the secret contents but for whatever reason openshift-oauth is throwing a fit:

Copying system trust bundle
I0117 15:24:30.133540       1 dynamic_serving_content.go:110] "Loaded a new cert/key pair" name="serving-cert::/var/config/system/secrets/v4-0-config-system-serving-cert/tls.crt::/var/config/system/secrets/v4-0-config-system-serving-cert/tls.key"
F0117 15:24:30.133791       1 cmd.go:56] failed to load SNI cert and key: tls: failed to find PEM block with type ending in "PRIVATE KEY" in key input after skipping PEM blocks of the following types: [CERTIFICATE CERTIFICATE CERTIFICATE]

[reggie-k]

My issue was the base64 padding. I have created the secret automatically by using ansible and then openshift could not decode it correctly. To understand the root cause, I created a secret in the ui with the chain, and then compared the resulted encoded string to the one that my automation had produced. They, indeed, were different. So I used rstrip=False in my playbook and then the string was encoded just fine, making all dependent operators happy. Did you chain manually?

[nate-duke]

Did you chain manually?

Kinda? Our CA doesn't ship back root-last bundles so there's sometimes some wrangling needed. I think i've isolated this down to a line-ending/formatting issue as well as the rendered secret the oauth pod is using has this lovely little dingleberry in it:

-----END CERTIFICATE----------BEGIN RSA PRIVATE KEY-----

Now I just get to run down which piece of machinery to poke in the eye for that lovely creation. We're not doing the concatenation, something in one of the operators is.

EDIT: Adding a blank line to the end of the plaintext certificate bundle material in the secret holding the wildcard bundle for the default router fixed this. Ugh.