Hot reload of mtls cert is broken. KIC can't fetch secret.

[codingwwws]

Hi (I think this might be similar to #958 but not sure)

I am using a cert for mtls inside a Service resource with:

  annotations:
    konghq.com/client-cert: my-service-secret
    konghq.com/protocol: https

Im getting the same error:

level=error msg="resource processing failed: failed to fetch secret 'my-service/my-service-secret': Secret my-service/my-service-secret not found" GVK="/v1, Kind=Service" name=my-service namespace=my-service

The Secret is there in the cluster and things are working fine. However once is deleted and recreated (renewed) by cert manager, the Ingress controller goes into the above error and the upstream service that is expecting the cert is receiving none. Steps to reproduce: 1- Setup a working mtls cert 2- Delete the secret containing the cert 3- cert manager will renew the mtls cert, recreating the Secret (same name, just new uid and resourceVersion and the cert content is changed) 4- The above error starts and that route to that Service doesn't send the new mtls cert to the upstream

Steps to solve currently is to add a dummy change to some annotation to the Service resource. Then something at the service levels clean reloads the cert and the error goes away and the cert is reloaded correctly and sent to upstream. Or to restart the Kong and kong ingress controller pods. That also loads the new cert and the error is only in case of the hot reload scenario.

In effect the hot reload of this cert is broken. KIC 2.12.3 Kong 3.4.2