Adding sidecars mysqld_exporter cannot create pxc pods

[dalesia]

Report

I can successfully create pxc cluster when I do not add sidecars mysqld_exporter Adding sidecars mysqld_exporter cannot create pxc pods cluster

More about the problem

kubectl get all -n pxc

NAME                                                   READY   STATUS    RESTARTS      AGE
pod/cluster1-haproxy-0                                 1/2     Running   2 (76s ago)   7m17s
pod/percona-xtradb-cluster-operator-7d6b49c48c-n9cpr   1/1     Running   0             8m52s

NAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                 AGE
service/cluster1-haproxy                  ClusterIP   100.64.198.221   <none>        3306/TCP,3309/TCP,33062/TCP,33060/TCP   7m17s
service/cluster1-haproxy-replicas         ClusterIP   100.64.73.9      <none>        3306/TCP                                7m17s
service/cluster1-pxc                      ClusterIP   None             <none>        3306/TCP,33062/TCP,33060/TCP            7m17s
service/cluster1-pxc-unready              ClusterIP   None             <none>        3306/TCP,33062/TCP,33060/TCP            7m17s
service/percona-xtradb-cluster-operator   ClusterIP   100.64.244.152   <none>        443/TCP                                 8m42s

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/percona-xtradb-cluster-operator   1/1     1            1           8m52s

NAME                                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/percona-xtradb-cluster-operator-7d6b49c48c   1         1         1       8m52s

NAME                                READY   AGE
statefulset.apps/cluster1-haproxy   0/3     7m17s
statefulset.apps/cluster1-pxc       0/3     7m17s

percona-xtradb-cluster-operator logs: 2024-03-05T10:27:52.211Z INFO Waiting for HAProxy to be ready before smart update {"controller": "pxc-controller", "namespace": "pxc", "name": "cluster1", "reconcileID": "29d4bc1a-0928-4ea7-8408-bc6da726381f"}

Steps to reproduce

deploy/cr.yaml sidecars config:

  pxc:
    size: 3
    image: percona-xtradb-cluster:8.0.32-24.2
    autoRecovery: true
    sidecars: 
    - image: mysqld-exporter:v0.15.1
      name: mysqld_exporter
      env: 
        - name: EXPORTER_PASS
          valueFrom:
            secretKeyRef:
              name: cluster1-secrets 
              key: monitor      
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP      
        - name: DATA_SOURCE_NAME
          value: "monitor:$(EXPORTER_PASS)@tcp(127.0.0.1:3306)/"
      args: ["--web.listen-address=$(POD_IP):9104"]
      resources:
        requests:
          memory: 100M
          cpu: 100m
        limits:
          memory: 200M
          cpu: 200m      

Versions

1. Kubernetes v1.28.3+rke2r2
2. Operator 1.13.0
3. Database percona-xtradb-cluster:8.0.32-24.2

Anything else?

No response

[spron-in]

@dalesia can you please describe the haproxy pod kubectl describe ... and maybe get some logs from haproxy.

I'm curious what is happening there.