AES-consul-connector unable to update secret "ambassador-consul-connect" with TLS certificate in Openshift4.12

[LearnitRm]

Installed Ambassador 3.2 , Consul 1.15 and AES-consul-connect 3.2.0 on RedHat Openshift platform 4.12 AES consul connector is unable to update TLS certificate to secret "ambassador-consul-connect" Below is the error : time="2023-07-10 03:30:17.6954" level=info msg="Watching CA leaf for ambassador\n" func=github.com/datawire/apro/v3/cmd/consul_connect_integration.Run file="github.com/datawire/apro/v3/cmd/consul_connect_integration/main.go:158" CMD=consul_connect_integration PID=1 time="2023-07-10 03:30:17.7850" level=info msg="caFingerprint A976D4187F529AFE319AE8067ECA1BA26F53CED3" func=github.com/datawire/apro/v3/cmd/consul_connect_integration.Run file="github.com/datawire/apro/v3/cmd/consul_connect_integration/main.go:211" CMD=consul_connect_integration PID=1 time="2023-07-10 03:30:17.7851" level=info msg="leafFingerprint C7E510CDA1AB40AAB1D3AAF150F624CC59052B1E" func=github.com/datawire/apro/v3/cmd/consul_connect_integration.Run file="github.com/datawire/apro/v3/cmd/consul_connect_integration/main.go:212" CMD=consul_connect_integration PID=1 time="2023-07-10 03:30:17.7852" level=info msg="leafCert 5A59A40B6C5276C4E3F4A8475CE72110F7B19966 notBefore 2023-07-10 02:44:20 +0000 UTC notAfter 2023-07-13 02:44:20 +0000 UTC" func=github.com/datawire/apro/v3/cmd/consul_connect_integration.Run file="github.com/datawire/apro/v3/cmd/consul_connect_integration/main.go:215" CMD=consul_connect_integration PID=1 time="2023-07-10 03:30:21.0892" level=error msg="Could not update TLS certificate secret ambassador-consul-connect.ambassador: &exec.ExitError{ProcessState:(*os.ProcessState)(0xc000940438), Stderr:[]uint8(nil)}" func=github.com/datawire/apro/v3/cmd/consul_connect_integration.Run file="github.com/datawire/apro/v3/cmd/consul_connect_integration/main.go:221" CMD=consul_connect_integration PID=1

Causing the TLS Expired Certificate Error while accessing API through the Ambassador LB IP , as the AES-consul-connector is unable to update the new TLS cert generated to the "ambassador-consul-connect" secret.

Expected behavior AES-consul-connect should update the secret "ambasador-consul-connect" with new TLS certs generated and inject into edge-stack. Should be able to git API url successfully.

Versions (please complete the following information):

• Ambassador: [3.2.0
• Kubernetes environment [ Openshift 4.12 , K8S 1.25 ]

• Consul [e.g. 1.15]

  Actions tried :

  1. Deleted the secret "amabassador-consul-connect" , rolled out restart on AES-consul-connect - couldnt fix the issue.
  2. Installed ambassador 3.6 - Couldnt fix the issue.
  3. Verified all RBAC policies etc - everything in place.

[cindymullins-dw]

Hi @LearnitRm, not sure if this was the same case as previously reported - was this fixed by including certain user roles, ie updating the apiGroup in cluster role from apiGroup:[ “”] to apiGroups: [“rbac.authorization.k8s.io”]?

[LearnitRm]

We are still seeing the issues after it updated the secret and 3 days today cert expired. We see the issue again

[dmaclaury]

I've followed the guide step for step in a clean local environment, and experience the same issue described here.

k logs -n ambassador ambassador-consul-connect-integration-b4c48448-9wmpk
time="2024-03-27 20:32:19.0310" level=info msg="/usr/bin/python3 /ambassador/kubewatch.py --debug failed with exit status 1\n\n" func=github.com/emissary-ingress/emissary/v3/pkg/environment.EnvironmentSetupEntrypoint file="github.com/emissary-ingress/emissary/v3@v3.9.1/pkg/environment/helper.go:45" CMD=consul_connect_integration PID=1
time="2024-03-27 20:32:19.1188" level=info msg="Starting Consul Connect Integration" func=github.com/datawire/apro/v3/cmd/consul_connect_integration.Run file="github.com/datawire/apro/v3/cmd/consul_connect_integration/main.go:132" CMD=consul_connect_integration PID=1 consul_host=10.42.0.44 consul_port=8500 version=3.9.1
time="2024-03-27 20:32:19.1202" level=info msg="Watching CA leaf for ambassador\n" func=github.com/datawire/apro/v3/cmd/consul_connect_integration.Run file="github.com/datawire/apro/v3/cmd/consul_connect_integration/main.go:158" CMD=consul_connect_integration PID=1
time="2024-03-27 20:32:19.2296" level=info msg="caFingerprint 9B303457AFBCB5BC94D19EC5592700043EB91095" func=github.com/datawire/apro/v3/cmd/consul_connect_integration.Run file="github.com/datawire/apro/v3/cmd/consul_connect_integration/main.go:211" CMD=consul_connect_integration PID=1
time="2024-03-27 20:32:19.2300" level=info msg="leafFingerprint 5B1211F523FC3F673120C427F50109C3D86EEBC4" func=github.com/datawire/apro/v3/cmd/consul_connect_integration.Run file="github.com/datawire/apro/v3/cmd/consul_connect_integration/main.go:212" CMD=consul_connect_integration PID=1
time="2024-03-27 20:32:19.2307" level=info msg="leafCert 821B1D7C247B67F3649F43ACED88217DA2AEE8FB notBefore 2024-03-27 20:31:19 +0000 UTC notAfter 2024-03-30 20:31:19 +0000 UTC" func=github.com/datawire/apro/v3/cmd/consul_connect_integration.Run file="github.com/datawire/apro/v3/cmd/consul_connect_integration/main.go:215" CMD=consul_connect_integration PID=1
time="2024-03-27 20:32:19.3230" level=error msg="Could not update TLS certificate secret ambassador-consul-connect.ambassador: &exec.Error{Name:\"kubectl\", Err:(*errors.errorString)(0x5ee8ab0)}" func=github.com/datawire/apro/v3/cmd/consul_connect_integration.Run file="github.com/datawire/apro/v3/cmd/consul_connect_integration/main.go:221" CMD=consul_connect_integration PID=1