Closed pfragoso closed 10 months ago
@pfragoso out of curiosity, how is the echoserver
deployed? I just want to make sure there isn't anything strange there, but I think that if the echoserver
is pod networked, then only the VXLAN addresses should be shown. @fasaxc you might know better, but does that sound right to you?
It's a normal k8s deployment + ingress (without hostNetwork or etc).
Here's the current spec applied:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
name: echoserver
namespace: echoserver
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: echoserver
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: echoserver
spec:
containers:
- image: gcr.io/kubernetes-e2e-test-images/echoserver:2.2
imagePullPolicy: Always
name: echoserver
ports:
- containerPort: 8080
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
Hm, interesting.
I don't think that the connection would succeed if the destination we responding with an IP address that didn't match the original conntrack entry on the host.
I wonder if it has to do with how the X-Real-IP header is populated by the server. Could you repeat the experiment and use tcpdump
on the client node to see what the actual IP address reported on the packet is?
What does the X-Real-IP header tell you? Is it the source IP address that the backing pod (or Traefik) saw? If so then that will depend on how the load balancing happened to get routed. If traffic goes
External -> Node A -> pod on node A
Then the pod will see the external IP.
If it goes
External -> Node A -> pod on node B
then there's an extra SNAT (done by kube-proxy).
Adding Traefik into the mix may change things again since there's another layer.
Trying calico as an alternative to weave I've bumped into some odd behavior regarding the addresses used.
The current setup is a normal EKS cluster with an ALB pointing to traefik as an ingress controller running an echo server to see the requests.
Running the latest tigera operator (3.25.0) with
Single node pool
traefik pods IP addresses and node information
running requests to the echoserver
Sometimes a node can reply using the host or the vxlan address. X-Real-Ip=10.138.12.108 X-Real-Ip=172.16.85.64
Expected Behavior
Not sure if this is expected but I would assume we should only see from one source
Your Environment