connect: Communication between pods of the same consul service

[Poweranimal]

Question

Hi there,

How can two or more Pods that belong to the same consul connect service communicate between each other?

Example: StatefulSet with 2 replicas, connect-inject enabled and transparent proxy enabled. The consul registered port is 8000. pod-0 sends a request to pod-1 (e.g. wget -qO- http://pod-1.pod.default:8000).

Context: StatefulSet services like minio have a gossip communication between their Pods on the consul registered service port. For that reason they must be able to communicate between each other.

Example Setup

1. Deploy consul with connect enabled and transparent proxy enabled.
2. Deploy the k8s manifest defined below
3. Run: kubectl exec nginx-0 -- curl -v http://nginx-1.nginx.default:8000

Expected Result: The curl command should return Hello world!

Actual Result: The curl command returns an empty response.

k8s Manifest

```yaml apiVersion: consul.hashicorp.com/v1alpha1 kind: ServiceDefaults metadata: name: nginx spec: protocol: http transparentProxy: dialedDirectly: true --- apiVersion: v1 kind: Service metadata: name: nginx spec: type: ClusterIP ports: - name: http port: 8000 targetPort: 8000 selector: app.kubernetes.io/instance: nginx app.kubernetes.io/name: nginx component: nginx --- apiVersion: v1 kind: ServiceAccount metadata: name: nginx --- apiVersion: v1 kind: ConfigMap metadata: name: nginx data: default.conf.template: | server { listen 8000; location / { return 200 'Hello world!'; add_header Content-Type text/plain; } } --- apiVersion: apps/v1 kind: StatefulSet metadata: name: nginx spec: selector: matchLabels: app.kubernetes.io/instance: nginx app.kubernetes.io/name: nginx component: nginx serviceName: nginx replicas: 2 updateStrategy: type: RollingUpdate template: metadata: name: nginx annotations: consul.hashicorp.com/connect-inject: "true" labels: app.kubernetes.io/instance: nginx app.kubernetes.io/name: nginx component: nginx spec: serviceAccountName: nginx containers: - name: nginx image: docker.io/nginx:1.21.6 imagePullPolicy: IfNotPresent ports: - containerPort: 8000 name: http protocol: TCP volumeMounts: - mountPath: /etc/nginx/templates name: config readOnly: true volumes: - name: config configMap: name: nginx ```

[david-yu]

Hi @Poweranimal, We do have a field within the the ServiceDefaults CRD called DialedDirectly (link) which allows you to dial the proxies directly for a given Service. Could you try setting the DialedDirectly config to true?

[Poweranimal]

Hi @david-yu , I enabled this mode. However, this only seems to allow direct IP address dialing from another consul service to the target consul service. It seems to have no effect when it comes to communication between Pods that belong to the same consul service.

[david-yu]

Apologies, I had neglected to glance at the first part of your k8s manifest as I did not expand it. One thing to try is also to use some annotations on the pods themselves to exclude traffic redirection on certain ports. In this it seems you may want to exclude traffic re-direction based on inbound request to port 8000.

https://www.consul.io/docs/connect/transparent-proxy#traffic-redirection-configuration

template:
    metadata:
      name: nginx
      annotations:
        consul.hashicorp.com/connect-inject: "true"
        consul.hashicorp.com/transparent-proxy-exclude-inbound-ports: "8000"

[Poweranimal]

Though this would also bypass features like mTLS and ACL. I’d like to keep my traffic secured by the mTLS of consul connect. I guess services like minio or redis in HA mode might share confidential data between each other that should be protected by TLS encryption.

[Poweranimal]

@david-yu I assume my presented use case is not supported by consul connect?

[whiskeysierra]

I've been told by our support manager that this isn't supported right now.

Our use case is actually HashiCorp Vault, which also needs to securely talk to itself (to proxy requests to the current master).

Same applies to Redis in sentinel mode.

[justin-sto]

I know that this has been open for a while, but is there a fix for that issue? We are facing the same issue with Elasticsearch, where e.g. the master pods need to communicate with each other to elect a leader. Would appreciate feedback on that @david-yu

Thanks in advance :)