traffic from IPv4Addres and IPv4VXLANTunnelAddr

[pfragoso]

Trying calico as an alternative to weave I've bumped into some odd behavior regarding the addresses used.

The current setup is a normal EKS cluster with an ALB pointing to traefik as an ingress controller running an echo server to see the requests.

Running the latest tigera operator (3.25.0) with

installation:
  kubernetesProvider: "EKS"
  registry: quay.io
  cni:
    type: Calico
    ipam:
      type: Calico
  calicoNetwork:
    bgp: Disabled

Single node pool

ip-10-138-10-90.eu-west-2.compute.internal
                    projectcalico.org/IPv4Address: 10.138.10.90/24
                    projectcalico.org/IPv4VXLANTunnelAddr: 172.16.232.192
ip-10-138-11-11.eu-west-2.compute.internal
                    projectcalico.org/IPv4Address: 10.138.11.11/24
                    projectcalico.org/IPv4VXLANTunnelAddr: 172.16.50.128
ip-10-138-11-187.eu-west-2.compute.internal
                    projectcalico.org/IPv4Address: 10.138.11.187/24
                    projectcalico.org/IPv4VXLANTunnelAddr: 172.16.66.129
ip-10-138-12-108.eu-west-2.compute.internal
                    projectcalico.org/IPv4Address: 10.138.12.108/24
                    projectcalico.org/IPv4VXLANTunnelAddr: 172.16.85.64

traefik pods IP addresses and node information

kube-system        traefik-public-cd85d884f-2kr6q  172.16.50.129    ip-10-138-11-11.eu-west-2.compute.internal    
kube-system        traefik-public-cd85d884f-7t6k6  172.16.232.198   ip-10-138-10-90.eu-west-2.compute.internal    
kube-system        traefik-public-cd85d884f-hbt9t  172.16.85.67     ip-10-138-12-108.eu-west-2.compute.internal   

running requests to the echoserver

while true; do curl -s https://echoserver | grep ^X-Real-Ip  && sleep 1; done
X-Real-Ip=172.16.85.64
X-Real-Ip=172.16.232.192
X-Real-Ip=10.138.10.90
X-Real-Ip=172.16.50.128
X-Real-Ip=172.16.50.128
X-Real-Ip=172.16.232.192
X-Real-Ip=172.16.85.64
X-Real-Ip=172.16.232.192
X-Real-Ip=172.16.50.128
X-Real-Ip=10.138.12.108
X-Real-Ip=172.16.232.192
X-Real-Ip=172.16.85.64
X-Real-Ip=172.16.50.128
X-Real-Ip=172.16.232.192
X-Real-Ip=172.16.232.192
X-Real-Ip=10.138.12.108
X-Real-Ip=172.16.85.64
X-Real-Ip=172.16.50.128
X-Real-Ip=172.16.50.128
X-Real-Ip=172.16.232.192
X-Real-Ip=10.138.12.108
X-Real-Ip=172.16.232.192
X-Real-Ip=172.16.85.64
X-Real-Ip=172.16.50.128
X-Real-Ip=172.16.50.128
X-Real-Ip=172.16.232.192
X-Real-Ip=172.16.232.192
X-Real-Ip=10.138.12.108
X-Real-Ip=172.16.85.64
X-Real-Ip=172.16.50.128
X-Real-Ip=172.16.50.128
X-Real-Ip=172.16.232.192
X-Real-Ip=172.16.85.64
X-Real-Ip=172.16.50.128
X-Real-Ip=172.16.232.192
X-Real-Ip=10.138.12.108
X-Real-Ip=172.16.50.128
X-Real-Ip=172.16.232.192

Sometimes a node can reply using the host or the vxlan address. X-Real-Ip=10.138.12.108 X-Real-Ip=172.16.85.64

Expected Behavior

Not sure if this is expected but I would assume we should only see from one source

Your Environment

• Calico version: 3.25.0
• Orchestrator version (e.g. kubernetes, mesos, rkt): EKS Kubernetes 1.22
• Operating System and version: AL2 amazon-eks-node-1.22-v20230105
• Link to your project (optional):

[mgleung]

@pfragoso out of curiosity, how is the echoserver deployed? I just want to make sure there isn't anything strange there, but I think that if the echoserver is pod networked, then only the VXLAN addresses should be shown. @fasaxc you might know better, but does that sound right to you?

[pfragoso]

It's a normal k8s deployment + ingress (without hostNetwork or etc).

Here's the current spec applied:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
  name: echoserver
  namespace: echoserver
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: echoserver
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: echoserver
    spec:
      containers:
      - image: gcr.io/kubernetes-e2e-test-images/echoserver:2.2
        imagePullPolicy: Always
        name: echoserver
        ports:
        - containerPort: 8080
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

[caseydavenport]

Hm, interesting.

I don't think that the connection would succeed if the destination we responding with an IP address that didn't match the original conntrack entry on the host.

I wonder if it has to do with how the X-Real-IP header is populated by the server. Could you repeat the experiment and use tcpdump on the client node to see what the actual IP address reported on the packet is?

[fasaxc]

What does the X-Real-IP header tell you? Is it the source IP address that the backing pod (or Traefik) saw? If so then that will depend on how the load balancing happened to get routed. If traffic goes

External -> Node A -> pod on node A

Then the pod will see the external IP.

If it goes

External -> Node A -> pod on node B

then there's an extra SNAT (done by kube-proxy).

Adding Traefik into the mix may change things again since there's another layer.