huntergregory
commented
2 months ago This HTTP experiment shows that for TCP, we similarly have double the number of flows as there are packets, yet we specify different trace_observation_point for the otherwise duplicate flow. We also do not have the ingress/egress labeling bug. That is, we have:
toolbox -> dep-http
- EGRESS, TO_STACK
- EGRESS, TO_NETWORK
- INGRESS, FROM_NETWORK
- INGRESS, TO_ENDPOINT
toolbox <- dep-http
- EGRESS, TO_STACK
- EGRESS, TO_NETWORK
- INGRESS, FROM_NETWORK
- INGRESS, TO_ENDPOINT
Additional Setup
- Below
http-dep:apiVersion: apps/v1 kind: Deployment metadata: name: dep-http spec: replicas: 1 selector: matchLabels: pod: http template: metadata: labels: pod: http spec: containers: - command: - /agnhost - serve-hostname - --http=true - --port - "80" image: k8s.gcr.io/e2e-test-images/agnhost:2.33 imagePullPolicy: IfNotPresent name: cont-80-tcp ports: - containerPort: 80 name: serve-80-tcp protocol: TCP nodeSelector: kubernetes.io/os: linux vm: "1" - Label the node which does NOT have
toolbox-podon it withkubectl label node <name> vm=1.
HTTP Experiment
Pods, Nodes, and IPs
NAME READY STATUS RESTARTS AGE IP NODE
dep-http-84f64997bd-j622q 1/1 Running 0 16m 10.224.1.166 aks-nodepool1-32351195-vmss000001
toolbox-pod 1/1 Running 0 44h 10.224.0.106 aks-nodepool1-32351195-vmss000000Experiment
Apply these iptables rules to each node to count TCP packets between the two Pods:
iptables-nft -I FORWARD -s 10.224.0.106 -d 10.224.1.166 --protocol tcp --dport 80
iptables-nft -I FORWARD -d 10.224.0.106 -s 10.224.1.166 --protocol tcp --sport 80Curl the http server:
k exec toolbox-pod -- curl 10.224.1.166:80Before curling, start observing flows:
$ hubble observe flows --protocol tcp -f | grep "toolbox-pod\|dep-http"
Aug 7 19:39:39.821: toolbox-pod:36600 (ID:1679) -> dep-http-84f64997bd-j622q:80 (ID:23273) to-stack FORWARDED (TCP Flags: SYN:true)
Aug 7 19:39:39.821: toolbox-pod:36600 (ID:1679) -> dep-http-84f64997bd-j622q:80 (ID:23273) to-network FORWARDED (TCP Flags: SYN:true)
Aug 7 19:39:39.823: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) from-network FORWARDED (TCP Flags: SYN:true ACK:true)
Aug 7 19:39:39.823: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-endpoint FORWARDED (TCP Flags: SYN:true ACK:true)
Aug 7 19:39:39.824: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-stack FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.824: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-network FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.831: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-stack FORWARDED (TCP Flags: PSH:true ACK:true)
Aug 7 19:39:39.831: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-network FORWARDED (TCP Flags: PSH:true ACK:true)
Aug 7 19:39:39.832: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) from-network FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.832: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-endpoint FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.832: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) from-network FORWARDED (TCP Flags: PSH:true ACK:true)
Aug 7 19:39:39.832: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-endpoint FORWARDED (TCP Flags: PSH:true ACK:true)
Aug 7 19:39:39.832: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-stack FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.832: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-network FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.844: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-stack FORWARDED (TCP Flags: FIN:true ACK:true)
Aug 7 19:39:39.844: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-network FORWARDED (TCP Flags: FIN:true ACK:true)
Aug 7 19:39:39.845: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) from-network FORWARDED (TCP Flags: FIN:true ACK:true)
Aug 7 19:39:39.845: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-endpoint FORWARDED (TCP Flags: FIN:true ACK:true)
Aug 7 19:39:39.845: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-stack FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.845: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-network FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.825: toolbox-pod:36600 (ID:1679) -> dep-http-84f64997bd-j622q:80 (ID:23273) from-network FORWARDED (TCP Flags: SYN:true)
Aug 7 19:39:39.825: toolbox-pod:36600 (ID:1679) -> dep-http-84f64997bd-j622q:80 (ID:23273) to-endpoint FORWARDED (TCP Flags: SYN:true)
Aug 7 19:39:39.825: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-stack FORWARDED (TCP Flags: SYN:true ACK:true)
Aug 7 19:39:39.825: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-network FORWARDED (TCP Flags: SYN:true ACK:true)
Aug 7 19:39:39.826: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) from-network FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.826: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-endpoint FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.833: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) from-network FORWARDED (TCP Flags: PSH:true ACK:true)
Aug 7 19:39:39.834: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-endpoint FORWARDED (TCP Flags: PSH:true ACK:true)
Aug 7 19:39:39.834: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-stack FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.834: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-network FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.834: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) from-network FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.834: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-endpoint FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.846: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) from-network FORWARDED (TCP Flags: FIN:true ACK:true)
Aug 7 19:39:39.846: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-endpoint FORWARDED (TCP Flags: FIN:true ACK:true)
Aug 7 19:39:39.846: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-stack FORWARDED (TCP Flags: FIN:true ACK:true)
Aug 7 19:39:39.846: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-network FORWARDED (TCP Flags: FIN:true ACK:true)
Aug 7 19:39:39.847: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) from-network FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.847: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-endpoint FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.834: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-stack FORWARDED (TCP Flags: PSH:true ACK:true)
Aug 7 19:39:39.834: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-network FORWARDED (TCP Flags: PSH:true ACK:true)The packets sent/received were:
Node 0:
root@aks-nodepool1-32351195-vmss000000:/# iptables -vnL FORWARD | grep 10.224.1.166
# Warning: iptables-legacy tables present, use iptables-legacy to see them
0 0 tcp -- * * 10.224.1.166 10.224.0.106 tcp spt:80
0 0 tcp -- * * 10.224.0.106 10.224.1.166 tcp dpt:80
root@aks-nodepool1-32351195-vmss000000:/#
root@aks-nodepool1-32351195-vmss000000:/# iptables -vnL FORWARD | grep 10.224.1.166
# Warning: iptables-legacy tables present, use iptables-legacy to see them
4 358 tcp -- * * 10.224.1.166 10.224.0.106 tcp spt:80
6 396 tcp -- * * 10.224.0.106 10.224.1.166 tcp dpt:80Node 1:
root@aks-nodepool1-32351195-vmss000001:/# iptables -vnL FORWARD | grep 10.224.1.166
# Warning: iptables-legacy tables present, use iptables-legacy to see them
0 0 tcp -- * * 10.224.1.166 10.224.0.106 tcp spt:80
0 0 tcp -- * * 10.224.0.106 10.224.1.166 tcp dpt:80
root@aks-nodepool1-32351195-vmss000001:/#
root@aks-nodepool1-32351195-vmss000001:/# iptables -vnL FORWARD | grep 10.224.1.166
# Warning: iptables-legacy tables present, use iptables-legacy to see them
4 358 tcp -- * * 10.224.1.166 10.224.0.106 tcp spt:80
6 396 tcp -- * * 10.224.0.106 10.224.1.166 tcp dpt:80
For a Retina cluster with Hubble enabled, when a Pod makes a DNS request (to the kube-dns service), if the request lands on a coredns Pod on another Node, then running hubble observe flows will produce double the necessary number of flows. Additionally, there seems to be a small bug in labeling the traffic as ingress versus egress.
In the DNS experiment below, there exist identical copies of DNS flows (looking at JSON format), resulting in 48 DNS flows between
toolbox-podandcoredns-5fc6484dd7-bnzlf, while there are 24 UDP packets when adding packets leaving Node 1 (src=toolbox, dst=coredns), entering Node 2 (src=toolbox, dst=coredns), leaving Node 2 (dst=toolbox, src=coredns), then entering Node 1 (dst=toolbox, src=coredns).Also in this experiment, there seems to be a bug as exemplified below. A single packet travels from
toolboxtocoredns-5fc6484dd7-bnzlf, and a single packet is sent back. Yet there is this discrepancy in egress vs. ingress counts. Presumably, the first and second flows are traced on one Node, and the third and fourth flows are traced on the other Node.toolbox -> bnzlf
toolbox <- bnzlf
Setup
mcr.microsoft.com/containernetworking/retina-agent-enterprise:v0.0.8mcr.microsoft.com/oss/cilium/hubble-relay:v1.15.0toolbox-pod:DNS Experiment
Pods, Nodes, and IPs
We will focus on
coredns-5fc6484dd7-bnzlf, which is on a different Node thantoolbox-pod.Experiment
Apply these iptables rules to each node to count UDP packets between the two Pods:
Curl bing.com
Before curling, start observing flows (focus only on the flows to/from
coredns-5fc6484dd7-bnzlf):The packets sent/received were:
Node 0:
Node 1:
When inspecting JSON output of the flows (
hubble observe -ojson), we see the flows are duplicated and there exists the ingress/egress labeling bug described at the top. Here is an example looking at thetester.default.svc.cluster.localquery: