AWX-Operator's rediness and liveness probs are failing

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that AWX is open source software provided for free and that I might not receive a timely response.
[X] I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

Liveness and rediness probs filing from awx-operator and keep restarting the awx-manager container from k8s pod

AWX version

23.9.0

Select the relevant components

[X] UI
[ ] UI (tech preview)
[X] API
[X] Docs
[X] Collection
[X] CLI
[ ] Other

Installation method

kubernetes

Modifications

Ansible version

2.12.8

Operating system

centos 8

Web browser

Chrome

Steps to reproduce

Deploy awx-operator version 2.12.2

Expected results

Pod shoud successfully deploy without any errors

Actual results

[ec2-user@ip-192-168-31-118 awx-21]$ k logs awx-operator-controller-manager-768c7bcbdd-knnmk -n awx-21 -c awx-manager -f {"level":"info","ts":"2024-04-08T09:32:52Z","logger":"cmd","msg":"Version","Go Version":"go1.20.12","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v1.34.0","commit":"d26c43bf94960d292152862a6685696be33190fb"} {"level":"info","ts":"2024-04-08T09:32:52Z","logger":"cmd","msg":"Watching namespaces","namespaces":["awx-21"]} {"level":"info","ts":"2024-04-08T09:32:52Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWX_AWX_ANSIBLE_COM","default":2} {"level":"info","ts":"2024-04-08T09:32:52Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXBACKUP_AWX_ANSIBLE_COM","default":2} {"level":"info","ts":"2024-04-08T09:32:52Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXRESTORE_AWX_ANSIBLE_COM","default":2} {"level":"info","ts":"2024-04-08T09:32:52Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXMESHINGRESS_AWX_ANSIBLE_COM","default":2} {"level":"info","ts":"2024-04-08T09:32:52Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWX"} {"level":"info","ts":"2024-04-08T09:32:52Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWXBackup"} {"level":"info","ts":"2024-04-08T09:32:52Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWXRestore"} {"level":"info","ts":"2024-04-08T09:32:52Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1alpha1","Options.Kind":"AWXMeshIngress"} {"level":"info","ts":"2024-04-08T09:32:52Z","logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"} {"level":"info","ts":"2024-04-08T09:32:52Z","logger":"apiserver","msg":"Starting to serve metrics listener","Address":"localhost:5050"} {"level":"info","ts":"2024-04-08T09:32:52Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"} {"level":"info","ts":"2024-04-08T09:32:52Z","msg":"starting server","kind":"health probe","addr":"[::]:6789"} {"level":"info","ts":"2024-04-08T09:32:52Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":"127.0.0.1:8080","secure":false} I0408 09:32:52.522017 7 leaderelection.go:250] attempting to acquire leader lease awx-21/awx-operator... I0408 09:33:17.733681 7 leaderelection.go:260] successfully acquired lease awx-21/awx-operator {"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting EventSource","controller":"awx-controller","source":"kind source: unstructured.Unstructured"} {"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting EventSource","controller":"awxbackup-controller","source":"kind source: unstructured.Unstructured"} {"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting Controller","controller":"awxbackup-controller"} {"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting Controller","controller":"awx-controller"} {"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting EventSource","controller":"awxmeshingress-controller","source":"kind source: unstructured.Unstructured"} {"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting Controller","controller":"awxmeshingress-controller"} {"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting EventSource","controller":"awxrestore-controller","source":"kind source: unstructured.Unstructured"} {"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting Controller","controller":"awxrestore-controller"} {"level":"error","ts":"2024-04-08T09:33:17Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind \"AWXMeshIngress\" in group \"awx.ansible.com\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:49\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:50\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"} {"level":"error","ts":"2024-04-08T09:33:17Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind \"AWXMeshIngress\" in group \"awx.ansible.com\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"} {"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting workers","controller":"awxbackup-controller","worker count":8} {"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting workers","controller":"awx-controller","worker count":8} {"level":"info","ts":"2024-04-08T09:33:17Z","msg":"Starting workers","controller":"awxrestore-controller","worker count":8} {"level":"error","ts":"2024-04-08T09:33:27Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind \"AWXMeshIngress\" in group \"awx.ansible.com\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"} {"level":"error","ts":"2024-04-08T09:33:37Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind \"AWXMeshIngress\" in group \"awx.ansible.com\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"} {"level":"error","ts":"2024-04-08T09:33:47Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind \"AWXMeshIngress\" in group \"awx.ansible.com\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"} {"level":"error","ts":"2024-04-08T09:33:57Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind \"AWXMeshIngress\" in group \"awx.ansible.com\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"} {"level":"error","ts":"2024-04-08T09:34:07Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind \"AWXMeshIngress\" in group \"awx.ansible.com\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"} {"level":"error","ts":"2024-04-08T09:34:17Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind \"AWXMeshIngress\" in group \"awx.ansible.com\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"} {"level":"error","ts":"2024-04-08T09:34:27Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind \"AWXMeshIngress\" in group \"awx.ansible.com\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"} {"level":"error","ts":"2024-04-08T09:34:37Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind \"AWXMeshIngress\" in group \"awx.ansible.com\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"} {"level":"error","ts":"2024-04-08T09:34:47Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind \"AWXMeshIngress\" in group \"awx.ansible.com\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"} {"level":"error","ts":"2024-04-08T09:34:57Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind \"AWXMeshIngress\" in group \"awx.ansible.com\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"} {"level":"error","ts":"2024-04-08T09:35:07Z","logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"AWXMeshIngress.awx.ansible.com","error":"failed to get restmapping: no matches for kind \"AWXMeshIngress\" in group \"awx.ansible.com\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:63\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:73\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/loop.go:74\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.28.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(Kind).Start.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56"} {"level":"error","ts":"2024-04-08T09:35:17Z","msg":"Could not wait for Cache to sync","controller":"awxmeshingress-controller","error":"failed to wait for awxmeshingress-controller caches to sync: timed out waiting for cache to be synced for Kind unstructured.Unstructured","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:203\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:208\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:234\nsigs.k8s.io/controller-runtime/pkg/manager.(runnableGroup).reconcile.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/manager/runnable_group.go:223"} {"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Stopping and waiting for non leader election runnables"} {"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Stopping and waiting for leader election runnables"} {"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"awxrestore-controller"} {"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"awx-controller"} {"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"awxbackup-controller"} {"level":"info","ts":"2024-04-08T09:35:17Z","msg":"All workers finished","controller":"awxrestore-controller"} {"level":"info","ts":"2024-04-08T09:35:17Z","msg":"All workers finished","controller":"awx-controller"} {"level":"info","ts":"2024-04-08T09:35:17Z","msg":"All workers finished","controller":"awxbackup-controller"} {"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Stopping and waiting for caches"} {"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Stopping and waiting for webhooks"} {"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Stopping and waiting for HTTP servers"} {"level":"info","ts":"2024-04-08T09:35:17Z","logger":"controller-runtime.metrics","msg":"Shutting down metrics server with timeout of 1 minute"} {"level":"info","ts":"2024-04-08T09:35:17Z","msg":"shutting down server","kind":"health probe","addr":"[::]:6789"} {"level":"info","ts":"2024-04-08T09:35:17Z","msg":"Wait completed, proceeding to shutdown the manager"} {"level":"error","ts":"2024-04-08T09:35:17Z","logger":"cmd","msg":"Proxy or operator exited with error.","error":"failed to wait for awxmeshingress-controller caches to sync: timed out waiting for cache to be synced for Kind unstructured.Unstructured","stacktrace":"github.com/operator-framework/ansible-operator-plugins/internal/cmd/ansible-operator/run.run\n\tansible-operator-plugins/internal/cmd/ansible-operator/run/cmd.go:261\ngithub.com/operator-framework/ansible-operator-plugins/internal/cmd/ansible-operator/run.NewCmd.func1\n\tansible-operator-plugins/internal/cmd/ansible-operator/run/cmd.go:81\ngithub.com/spf13/cobra.(Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115\ngithub.com/spf13/cobra.(Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039\nmain.main\n\tansible-operator-plugins/cmd/ansible-operator/main.go:40\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.12/x64/src/runtime/proc.go:250"} {"level":"error","ts":"2024-04-08T09:35:17Z","msg":"error received after stop sequence was engaged","error":"leader election lost","stacktrace":"sigs.k8s.io/controller-runtime/pkg/manager.(controllerManager).engageStopProcedure.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/manager/internal.go:490"}

Additional information

Events: Type Reason Age From Message

Normal Scheduled 21m default-scheduler Successfully assigned awx-21/awx-operator-controller-manager-768c7bcbdd-knnmk to ip-192-168-74-156.ec2.internal Normal Pulling 21m kubelet Pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0" Normal Pulled 20m kubelet Successfully pulled image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0" in 12.675458384s (12.675472904s including waiting) Normal Created 20m kubelet Created container kube-rbac-proxy Normal Started 20m kubelet Started container kube-rbac-proxy Normal Pulling 20m kubelet Pulling image "468681639671.dkr.ecr.us-east-1.amazonaws.com/concierto-uat-awx-operator:23.9.0.2" Normal Pulled 6m19s kubelet Successfully pulled image "468681639671.dkr.ecr.us-east-1.amazonaws.com/concierto-uat-awx-operator:23.9.0.2" in 14m35.572378533s (14m35.572392633s including waiting) Normal Created 5m29s kubelet Created container awx-manager Normal Started 5m27s kubelet Started container awx-manager Warning Unhealthy 2m11s (x3 over 2m51s) kubelet Liveness probe failed: Get "http://192.168.83.70:6789/healthz": dial tcp 192.168.83.70:6789: connect: connection refused Normal Killing 2m11s kubelet Container awx-manager failed liveness probe, will be restarted Warning Unhealthy 41s (x17 over 3m1s) kubelet Readiness probe failed: Get "http://192.168.83.70:6789/readyz": dial tcp 192.168.83.70:6789: connect: connection refused

ansible / awx