Consul does not ignore services with consul.hashicorp.com/service-ignore=true

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

I am deploying RabbitMQ and integrating it into a Consul service mesh. RabbitMQ ships with multiple services pointing towards it server. The consul-connect-inject-init init container is giving errors that there are multiple services pointing to the pod and I should set the consul.hashicorp.com/service-ignore=true label. All but one of the services has this label assigned but it's still giving the error.

2024-07-09T21:09:29.076Z [ERROR] There are multiple Consul services registered for this pod when there must only be one. Check if there are multiple Kubernetes services selecting this pod and add the label `consul.hashicorp.com/service-ignore: "true"` to all services except the one used by Consul for handling requests.
2024-07-09T21:09:29.076Z [ERROR] Timed out waiting for service registration: error="did not find correct number of services, found: 4, services: &{Node:0xc001095050 Services:[0xc000bf4ea0 0xc000bf50e0 0xc000bf5320 0xc000bf57a0]}"
2024-07-09T21:09:29.076Z [INFO]  consul-server-connection-manager: stopping
2024-07-09T21:09:29.078Z [INFO]  consul-server-connection-manager: ACL auth method logout succeeded

on aiocp2(rabbitmq) pcp-dev-sandbox/kbreit/crossplane on  main [!?]
➜ kubectl get svc --show-labels
NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                        AGE    LABELS
cluster          ClusterIP   172.30.237.192   <none>        15672/TCP,15692/TCP,5672/TCP   63d    app.kubernetes.io/component=rabbitmq,app.kubernetes.io/name=cluster,app.kubernetes.io/part-of=rabbitmq,consul.hashicorp.com/service-ignore=true
cluster-nodes    ClusterIP   None             <none>        4369/TCP,25672/TCP             63d    app.kubernetes.io/component=rabbitmq,app.kubernetes.io/name=cluster,app.kubernetes.io/part-of=rabbitmq,consul.hashicorp.com/service-ignore=true
cluster-server   ClusterIP   172.30.99.122    <none>        15672/TCP,15692/TCP,5672/TCP   7h3m   app.kubernetes.io/component=rabbitmq,app.kubernetes.io/name=cluster-server,app.kubernetes.io/part-of=rabbitmq

Reproduction Steps

Deploy RabbitMQ operator using the instructions at https://www.rabbitmq.com/kubernetes/operator/operator-overview. Alternatively, a statefulset could be deployed with a similar configuration for POC, but I haven't tested it with that setup. Deploy the operator controller with the following manifest. Note, some of this may be populated automatically.

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  annotations:
  labels:
    consul.hashicorp.com/service-ignore: "true"
  name: cluster
  namespace: rabbitmq
spec:
  delayStartSeconds: 30
  image: rabbitmq:3.13.1-management
  override:
    statefulSet:
      spec:
        template:
          metadata:
            annotations:
              consul.hashicorp.com/connect-inject: "true"
              consul.hashicorp.com/transparent-proxy: "true"
              k8s.v1.cni.cncf.io/networks: '[{ "name":"consul-cni","namespace": "rabbitmq"
                }]'
          spec:
            containers: []
            securityContext: {}
  persistence:
    storage: 10Gi
  rabbitmq: {}
  replicas: 1
  service:
    type: ClusterIP
  terminationGracePeriodSeconds: 604800
  tls: {}

Logs

Expected behavior

The pod should come up as it should not see the services which have this label.

Environment details

consul-k8s version: v1.18.2

values.yaml used to deploy the helm chart:

global:
name: consul
datacenter: aiocp2
image: registry.connect.redhat.com/hashicorp/consul:1.18.2-ubi
imageK8S: registry.connect.redhat.com/hashicorp/consul-k8s-control-plane:1.4.2-ubi
imageConsulDataplane: registry.connect.redhat.com/hashicorp/consul-dataplane:1.4.2-ubi
openshift:
enabled: true
tls:
enabled: true
caCert:
  secretName: consul-hubs-ca-cert
  secretKey: tls.crt
peering:
enabled: true
acls:
manageSystemACLs: true
ui:
enabled: true
ingress:
enabled: true
pathType: ImplementationSpecific
hosts:
  - host: consul-ui.apps.aiocp2.example.com
    paths:
      - ""
annotations: |
  "route.openshift.io/termination": "passthrough"
server:
replicas: 1
bootstrapExpect: 1
disruptionBudget:
enabled: true
maxUnavailable: 0
exposeService:
enabled: true
type: LoadBalancer
annotations: |
  metallb.universe.tf/loadBalancerIPs: 10.204.90.103
serverCert:
secretName: consul-hubs-server-cert
meshGateway:
enabled: true
wanAddress:
source: "Static"
static: "aiocp2-mesh-gateway.example.com"
service:
annotations: |
  metallb.universe.tf/loadBalancerIPs: 10.204.90.104
connect:
enabled: true
connectInject:
enabled: true
default: false
transparentProxy:
defaultEnabled: true
cni:
enabled: true
multus: true
cniBinDir: /var/lib/cni/bin
cniNetDir: /etc/kubernetes/cni/net.d
dns:
enabled: true
enableRedirection: true
type: LoadBalancer
annotations: |
metallb.universe.tf/loadBalancerIPs: 10.204.90.105

Kubernetes version: v1.28.10
Cloud Provider (If self-hosted, the Kubernetes provider utilized): OpenShift 4.15.17 on vSphere
Networking CNI plugin in use: Multus

hashicorp / consul-k8s