[init-database] - Issues to connect to the database when workload is meshed using Istio

[tchellomello]

Please confirm the following

• [X] I agree to follow this project's code of conduct.
• [X] I have checked the current issues for duplicates.
• [X] I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

With the feature introduced by the https://github.com/ansible/awx-operator/commit/ffba1b4712a0b03f1faedfa70e3a9ef0d443e4a6, a InitContainer named init-database was introduced to validate the database status by calling the wait-for-migrations script. I really liked this approach though, so we have a dedicated task for it.

Therefore, if the workload/namespace is meshed with Istio, the initContainer requires an special UID to be captured by the Istio, otherwise the initContainer will not be able to reach out to the database. Note that this issue does not happen with a regular pod, as the sidecar will get injected as expected, whereas on the initContainer there is no sidecar yet.

To fix the issue, we need to expose to this deployment the ability to override the securityContext to runAsUser.

To address this issue, I stopped the operator (to avoid reconciliation) and manually patch the awx-task deployment which then works as expected.

❯ kubectl get deploy awx-toca-task -o json | jq '.spec.template.spec.initContainers.[0].securityContext'
{
  "runAsUser": 1337
}

❯ kubectl images -n awx
[Summary]: 1 namespaces, 3 pods, 15 containers and 4 different images
+---------------------------------+-------------------------+--------------------------------+
|               Pod               |        Container        |             Image              |
+---------------------------------+-------------------------+--------------------------------+
| awx-toca-migration-24.2.0-kg4m6 | migration-job           | quay.io/ansible/awx:24.2.0     |
+---------------------------------+-------------------------+--------------------------------+
| awx-toca-task-5bdc6ff596-xsj4z  | redis                   | docker.io/redis:7              |
+                                 +-------------------------+--------------------------------+
|                                 | awx-toca-task           | quay.io/ansible/awx:24.2.0     |
+                                 +-------------------------+--------------------------------+
|                                 | awx-toca-ee             | quay.io/ansible/awx-ee:24.2.0  |
+                                 +-------------------------+--------------------------------+
|                                 | awx-toca-rsyslog        | quay.io/ansible/awx:24.2.0     |
+                                 +-------------------------+--------------------------------+
|                                 | istio-proxy             | docker.io/istio/proxyv2:1.21.1 |
+                                 +-------------------------+                                +
|                                 | (init) istio-validation |                                |
+                                 +-------------------------+--------------------------------+
|                                 | (init) init-database    | quay.io/ansible/awx:24.2.0     |
+                                 +-------------------------+--------------------------------+
|                                 | (init) init-receptor    | quay.io/ansible/awx-ee:24.2.0  |
+---------------------------------+-------------------------+--------------------------------+
| awx-toca-web-6f4657cbfb-gzhz5   | redis                   | docker.io/redis:7              |
+                                 +-------------------------+--------------------------------+
|                                 | awx-toca-web            | quay.io/ansible/awx:24.2.0     |
+                                 +-------------------------+                                +
|                                 | awx-toca-rsyslog        |                                |
+                                 +-------------------------+--------------------------------+
|                                 | istio-proxy             | docker.io/istio/proxyv2:1.21.1 |
+                                 +-------------------------+                                +
|                                 | (init) istio-validation |                                |
+                                 +-------------------------+--------------------------------+
|                                 | (init) init             | quay.io/ansible/awx-ee:24.2.0  |
+---------------------------------+-------------------------+--------------------------------+

❯ kubectl get pods -n awx
NAME                              READY   STATUS      RESTARTS   AGE
awx-toca-migration-24.2.0-kg4m6   0/1     Completed   0          61m
awx-toca-task-5bdc6ff596-xsj4z    5/5     Running     0          50m
awx-toca-web-6f4657cbfb-gzhz5     4/4     Running     0          53m

AWX Operator version

2.15.0

AWX version

24.2.0

Kubernetes platform

kubernetes

Kubernetes/Platform version

1.29

Modifications

yes

Steps to reproduce

1. Mesh the namespace using Istio
2. Deploy workload

Expected results

Work as expect

Actual results

Fails to connect to the database due to initContainer being unable to reach out to the network.

Additional information

N/A

Operator Logs

N/A

### Tasks

[RaceFPV]

+1 just ran into this same issue today, task never comes online due to init containers firing before the istio sidecar is online.

[mick1627]

Adding support for task_label to set sidecar.istio.io/inject to false could be a solution. There is a helm variable task_annotations for annotations but using annotation is deprecated https://istio.io/latest/docs/reference/config/annotations/#SidecarInject

[RaceFPV]

I don't think thats a reasonable solution as our end goal here is to run istio successfully while using awx-operator, which we can do now we just have to pause controller enforcement to do so until a proper patch is released.