V6 vxlanCrossSubNet does not create routes over vxlan in purely ipv6 env

          We ran into a similar issue when we tested Calico. Only difference in our setup is that we run IPv6 Single-Stack.

DNS was not working, but the underlying issue was deeper and general pod-to-pod traffic was not working between pods on different nodes.

Config looked as follows:

apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  annotations:
    meta.helm.sh/release-name: calico
    meta.helm.sh/release-namespace: tigera-operator
  creationTimestamp: "2024-10-18T07:58:09Z"
  finalizers:
  - operator.tigera.io/installation-controller
  - tigera.io/operator-cleanup
  - operator.tigera.io/apiserver-controller
  generation: 3
  labels:
    app.kubernetes.io/managed-by: Helm
    helm.toolkit.fluxcd.io/name: test-cluster-addons-calico
    helm.toolkit.fluxcd.io/namespace: provider
  name: default
  resourceVersion: "2351712"
  uid: 93644abd-7176-43ea-9c86-ee3b103b86b3
spec:
  calicoNetwork:
    bgp: Disabled
    hostPorts: Enabled
    ipPools:
    - allowedUses:
      - Workload
      - Tunnel
      blockSize: 122
      cidr: fd80:cafe::/104
      disableBGPExport: true
      encapsulation: VXLANCrossSubnet
      name: default-ipv6-ippool
      natOutgoing: Enabled
      nodeSelector: all()
    linuxDataplane: BPF
    multiInterfaceMode: None
    nodeAddressAutodetectionV4: {}
    nodeAddressAutodetectionV6:
      kubernetes: NodeInternalIP
    windowsDataplane: Disabled
  cni:
    ipam:
      type: Calico
    type: Calico
  controlPlaneReplicas: 2
  flexVolumePath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
  imagePullSecrets: []
  kubeletVolumePluginPath: /var/lib/kubelet
  kubernetesProvider: ""
  logging:
    cni:
      logFileMaxAgeDays: 30
      logFileMaxCount: 10
      logFileMaxSize: 100Mi
      logSeverity: Debug
  nodeUpdateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
  nonPrivileged: Disabled
  variant: Calico
---
apiVersion: projectcalico.org/v3
kind: FelixConfiguration
metadata:
  annotations:
    operator.tigera.io/bpfEnabled: "true"
  creationTimestamp: "2024-10-18T07:58:14Z"
  generation: 1
  name: default
  resourceVersion: "2351439"
  uid: 7d7bcc1d-ec32-4f17-9099-8b55d64a4dd2
spec:
  bpfConnectTimeLoadBalancing: TCP
  bpfEnabled: true
  bpfHostNetworkedNATWithoutCTLB: Enabled
  bpfLogLevel: ""
  floatingIPs: Disabled
  healthPort: 9099
  logSeverityScreen: Info
  reportingInterval: 0s
  vxlanVNI: 4096

The root cause in our setup seems to be that the VXLAN device is not used as routes are incorrectly set by Calico:

# ip -6 r
2001:db8::1 dev bpfin.cali metric 1024 pref medium
fd01:cafe::/64 dev eth0 proto ra metric 1024 mtu 65520 hoplimit 255 pref medium
fd80:cafe::5f:4281 dev cali3b862d89810 metric 1024 pref medium
fd80:cafe::5f:4282 dev cali9d386f561d2 metric 1024 pref medium
fd80:cafe::5f:4283 dev cali856dc622126 metric 1024 pref medium
blackhole fd80:cafe::5f:4280/122 dev lo proto 80 metric 1024 pref medium
fd80:cafe::a8:4d00/122 via fd01:cafe::f14c:9fa1:8496:5550 dev eth0 proto 80 metric 1024 onlink pref medium
fd85:cafe::a via 2001:db8::1 dev bpfin.cali src fd01:cafe::4aab:d761:d808:996 metric 1024 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev cali3b862d89810 proto kernel metric 256 pref medium
fe80::/64 dev cali9d386f561d2 proto kernel metric 256 pref medium
fe80::/64 dev bpfout.cali proto kernel metric 256 pref medium
fe80::/64 dev bpfin.cali proto kernel metric 256 pref medium
fe80::/64 dev cali856dc622126 proto kernel metric 256 pref medium
default dev eth0 proto static metric 1000 pref medium
default via fe80::ecee:eeff:feee:eeee dev eth0 proto ra metric 1024 expires 65136sec mtu 65520 hoplimit 255 pref medium

We were expecting fd80:cafe::a8:4d00/122 via fd01:cafe::f14c:9fa1:8496:5550 dev eth0 proto 80 metric 1024 onlink pref medium to go through the vxlan interface and not be directly routed to the second node (fd01...).

@matthewdupre Hopefully that helps. We are currently working around this issue by using flannel, but would prefer calico.

Originally posted by @trevex in https://github.com/projectcalico/calico/issues/8811#issuecomment-2421779071

projectcalico / calico

V6 vxlanCrossSubNet does not create routes over vxlan in purely ipv6 env #9403