Hammering EJBCA with request and lots of log

[smuda]

When the EJBCA does not answer the health check ("/ejbca/ejbca-rest-api/v1/certificate/status"), the operator keeps calling the endpoint and creates a log of log. It looks like roughly 10 requests per second and stays the same for hours.

Perhaps an exponential backoff would be a good thing or a lower frequency?

Log extract

``` 2024-05-02T14:24:56Z ERROR Reconciler error {"controller": "issuer", "controllerGroup": "ejbca-issuer.keyfactor.com", "controllerKind": "Issuer", "Issuer": {"name":"tls-server","namespace":"chainsaw-busy-peacock"}, "namespace": "chainsaw-busy-peacock", "name": "tls-server", "reconcileID": "4e9e661f-ffde-4613-9af5-eb2544ddda1a", "error": "healthcheck failed: Get \"https://ejbca.ejbca.svc/ejbca/ejbca-rest-api/v1/certificate/status\": read tcp 10.244.0.11:34674->10.96.129.198:443: read: connection reset by peer"} 2024-05-02T14:24:56Z ERROR Reconciler error {"controller": "issuer", "controllerGroup": "ejbca-issuer.keyfactor.com", "controllerKind": "Issuer", "Issuer": {"name":"tls-server","namespace":"chainsaw-busy-peacock"}, "namespace": "chainsaw-busy-peacock", "name": "tls-server", "reconcileID": "24a12b3e-6c0f-4e5f-a97f-ed962e7db613", "error": "healthcheck failed: Get \"https://ejbca.ejbca.svc/ejbca/ejbca-rest-api/v1/certificate/status\": EOF"} 2024-05-02T14:24:56Z ERROR Reconciler error {"controller": "issuer", "controllerGroup": "ejbca-issuer.keyfactor.com", "controllerKind": "Issuer", "Issuer": {"name":"tls-server","namespace":"chainsaw-busy-peacock"}, "namespace": "chainsaw-busy-peacock", "name": "tls-server", "reconcileID": "38033539-1fb4-4938-8174-c43e5c964281", "error": "healthcheck failed: Get \"https://ejbca.ejbca.svc/ejbca/ejbca-rest-api/v1/certificate/status\": read tcp 10.244.0.11:34690->10.96.129.198:443: read: connection reset by peer"} 2024-05-02T14:24:56Z ERROR Reconciler error {"controller": "issuer", "controllerGroup": "ejbca-issuer.keyfactor.com", "controllerKind": "Issuer", "Issuer": {"name":"tls-server","namespace":"chainsaw-busy-peacock"}, "namespace": "chainsaw-busy-peacock", "name": "tls-server", "reconcileID": "79df5813-8020-4622-a5c7-446cb80c982f", "error": "healthcheck failed: Get \"https://ejbca.ejbca.svc/ejbca/ejbca-rest-api/v1/certificate/status\": read tcp 10.244.0.11:34706->10.96.129.198:443: read: connection reset by peer"} 2024-05-02T14:24:56Z ERROR Reconciler error {"controller": "issuer", "controllerGroup": "ejbca-issuer.keyfactor.com", "controllerKind": "Issuer", "Issuer": {"name":"tls-server","namespace":"chainsaw-busy-peacock"}, "namespace": "chainsaw-busy-peacock", "name": "tls-server", "reconcileID": "b1aa1656-fb1b-4901-892a-2ccbe0ea85d7", "error": "healthcheck failed: Get \"https://ejbca.ejbca.svc/ejbca/ejbca-rest-api/v1/certificate/status\": read tcp 10.244.0.11:34722->10.96.129.198:443: read: connection reset by peer"} 2024-05-02T14:24:56Z ERROR Reconciler error {"controller": "issuer", "controllerGroup": "ejbca-issuer.keyfactor.com", "controllerKind": "Issuer", "Issuer": {"name":"tls-server","namespace":"chainsaw-busy-peacock"}, "namespace": "chainsaw-busy-peacock", "name": "tls-server", "reconcileID": "4137fb3d-dbdb-47a4-a5c4-93749fdd868b", "error": "healthcheck failed: Get \"https://ejbca.ejbca.svc/ejbca/ejbca-rest-api/v1/certificate/status\": read tcp 10.244.0.11:34734->10.96.129.198:443: read: connection reset by peer"} 2024-05-02T14:24:56Z ERROR Reconciler error {"controller": "issuer", "controllerGroup": "ejbca-issuer.keyfactor.com", "controllerKind": "Issuer", "Issuer": {"name":"tls-server","namespace":"chainsaw-busy-peacock"}, "namespace": "chainsaw-busy-peacock", "name": "tls-server", "reconcileID": "f204ac31-d9a9-41b4-a914-991797703a1b", "error": "healthcheck failed: Get \"https://ejbca.ejbca.svc/ejbca/ejbca-rest-api/v1/certificate/status\": read tcp 10.244.0.11:34748->10.96.129.198:443: read: connection reset by peer"} 2024-05-02T14:24:56Z ERROR Reconciler error {"controller": "issuer", "controllerGroup": "ejbca-issuer.keyfactor.com", "controllerKind": "Issuer", "Issuer": {"name":"tls-server","namespace":"chainsaw-busy-peacock"}, "namespace": "chainsaw-busy-peacock", "name": "tls-server", "reconcileID": "80a99637-d6e6-414d-b66b-347e93190f4f", "error": "healthcheck failed: Get \"https://ejbca.ejbca.svc/ejbca/ejbca-rest-api/v1/certificate/status\": read tcp 10.244.0.11:34752->10.96.129.198:443: read: connection reset by peer"} 2024-05-02T14:24:56Z ERROR Reconciler error {"controller": "issuer", "controllerGroup": "ejbca-issuer.keyfactor.com", "controllerKind": "Issuer", "Issuer": {"name":"tls-server","namespace":"chainsaw-busy-peacock"}, "namespace": "chainsaw-busy-peacock", "name": "tls-server", "reconcileID": "872820cd-e493-4b81-b5f6-4dd8de1365b3", "error": "healthcheck failed: Get \"https://ejbca.ejbca.svc/ejbca/ejbca-rest-api/v1/certificate/status\": read tcp 10.244.0.11:34760->10.96.129.198:443: read: connection reset by peer"} 2024-05-02T14:24:56Z ERROR Reconciler error {"controller": "issuer", "controllerGroup": "ejbca-issuer.keyfactor.com", "controllerKind": "Issuer", "Issuer": {"name":"tls-server","namespace":"chainsaw-busy-peacock"}, "namespace": "chainsaw-busy-peacock", "name": "tls-server", "reconcileID": "d27f3449-be31-4c8e-b25d-eb190e614585", "error": "healthcheck failed: Get \"https://ejbca.ejbca.svc/ejbca/ejbca-rest-api/v1/certificate/status\": read tcp 10.244.0.11:34762->10.96.129.198:443: read: connection reset by peer"} ```

[m8rmclaren]

Hi @smuda!

For context, the Health Check interface is used in the Issuer/ClusterIssuer controller to verify the connection/credentials to EJBCA. My understanding understanding up to now is that Reconcilers built using the controller-runtime project had exponential backoff built-in. I will do some research to determine if this is actually the case.

[m8rmclaren]

Hi @smuda

In the latest version of the ejbca-cert-manager-issuer, I upgraded from Kubebuilder go/v3 to go/v4 which upgraded the controller-runtime module (among many others).

While exponential backoff should have been implemented by controller-runtime in the previous version, I'm finding experimentally that it does work as expected in the newer version.

[smuda]

Well done!