ebfp mode does not support k8s option to not track dns to local node

mchtech commented 1 year ago

Expected Behavior

config doNotTrack GlobalNetworkPolicy: calico-node -bpf conntrack dump should not output related conntrack item

Current Behavior

calico-node -bpf conntrack dump outputs related conntrack items (seems that bpf still conntrack doNotTrack connections)

Possible Solution

Steps to Reproduce (for bugs)

FelixConfiguration for node $n

apiVersion: projectcalico.org/v3
kind: FelixConfiguration
metadata:
name: node.$n
spec:
defaultEndpointToHostAction: Return
floatingIPs: Disabled
ipipEnabled: false
# logFilePath: "/proc/1/fd/1"
prometheusGoMetricsEnabled: false
prometheusMetricsEnabled: true
prometheusProcessMetricsEnabled: false
prometheusWireGuardMetricsEnabled: false
reportingInterval: "0"
vxlanEnabled: false
# eBPF
bpfEnabled: true
bpfConnectTimeLoadBalancingEnabled: true
bpfExternalServiceMode: DSR
bpfKubeProxyIptablesCleanupEnabled: true
bpfKubeProxyEndpointSlicesEnabled: true
bpfMapSizeConntrack: 4194304 # 512000
failsafeInboundHostPorts: []
failsafeOutboundHostPorts: []

GlobalNetworkPolicy

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: node-local-dns-ebpf
spec:
applyOnForward: true
doNotTrack: true
types:
- Ingress
- Egress
ingress:
- action: Allow
  protocol: UDP
  source:
    nets:
      - 169.254.20.10/32
    ports:
      - 53
- action: Allow
  protocol: TCP
  source:
    nets:
      - 169.254.20.10/32
    ports:
      - 53
      - 9453
egress:
- action: Allow
  protocol: UDP
  destination:
    nets:
      - 169.254.20.10/32
    ports:
      - 53
- action: Allow
  protocol: TCP
  destination:
    nets:
      - 169.254.20.10/32
    ports:
      - 53
      - 9453

test pod $p on node $n
```
curl -v http://169.254.20.10:9453
```

calico pod on node $n

calico-node -bpf conntrack dump | grep ':9453'
# output
ConntrackKey{proto=6 169.254.20.10:9453 <-> 10.145.31.148:45700} -> Entry{Type:0, Created:68074914769170, LastSeen:68074915280661, Flags: 0x100 B-A Data: {A2B:{Bytes:0 Packets:0 Seqno:1974626082 SynSeen:true AckSeen:true FinSeen:true RstSeen:false Approved:false Opener:false Ifindex:0} B2A:{Bytes:0 Packets:0 Seqno:739750357 SynSeen:true AckSeen:true FinSeen:true RstSeen:false Approved:true Opener:true Ifindex:35} OrigDst:0.0.0.0 OrigSrc:0.0.0.0 OrigPort:0 OrigSPort:0 TunIP:0.0.0.0}} Age: 1.550954871s Active ago 1.55044338s CLOSED

Context

node-local-dns supports setup "iptables NOTRACK" rules to reduce conntrack items from Pod to Host local ip: https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/1024-nodelocal-cache-dns/README.md#iptables-notrack

-A PREROUTING -d 169.254.20.10/32 -p udp -m udp --dport 53 -j NOTRACK
-A PREROUTING -d 169.254.20.10/32 -p tcp -m tcp --dport 53 -j NOTRACK
-A OUTPUT -s 169.254.20.10/32 -p tcp -m tcp --sport 9453 -j NOTRACK
-A OUTPUT -d 169.254.20.10/32 -p tcp -m tcp --dport 9453 -j NOTRACK
-A OUTPUT -d 169.254.20.10/32 -p udp -m udp --dport 53 -j NOTRACK
-A OUTPUT -d 169.254.20.10/32 -p tcp -m tcp --dport 53 -j NOTRACK
-A OUTPUT -s 169.254.20.10/32 -p udp -m udp --sport 53 -j NOTRACK
-A OUTPUT -s 169.254.20.10/32 -p tcp -m tcp --sport 53 -j NOTRACK

udp conntrack timeout is 1min in calico-eBPF dataplane. (I don't find related docs to config calico-ebpf udp conntrack timeout) If app makes lots of short TCP connections (such as php app), dns udp 53 ebpf conntrack items will increase quickly but expire slowly.

Your Environment

Calico version: 3.26.1
Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes 1.26.7
Operating System and version: Oracle Linux 8.6 with elrepo kernel 5.16.11-1.el8.elrepo
Link to your project (optional): https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/1024-nodelocal-cache-dns/README.md#iptables-notrack

tomastigera commented 1 year ago

doNotTrack policy does not apply here and you cannot use it as a workaround for that missing feature which is k8s dependent and works for iptables mode only.

We will need to add a similar feature for the ebpf dataplane in calico.

Bisht13 commented 1 year ago

Can I take this up? @tomastigera

tomastigera commented 1 year ago

@Bisht13 sure have fun :)

sfudeus commented 11 months ago

Related to the use-case at least: https://github.com/projectcalico/calico/issues/4689

tomastigera commented 11 months ago

Closing as duplicate of #4689

projectcalico / calico