Return traffic to pods is dropped as INVALID by iptables

[rfranks-securenet]

I am running Calico in IPIP mode with BGP peering to the router.

Traffic that egresses from a pod reaches the router, and gets passed on to the destination. When the return traffic arrives, the BGP peering sends it to a different node, which then drops it within the iptables ruleset as invalid. Manually putting a rule in the KUBE-FORWARD chain that permits invalid traffic resolves it.

It may well be a misconfiguration or lack of understanding on how to set it up - if that's the case, apologies. I've looked in the documentation to try to find a resolution, but not had much luck.

Expected Behavior

Pod traffic should be able to be routed back to the pod even if ingressing via a different node

Current Behavior

Pod return traffic is dropped as invalid on ingress

Steps to Reproduce (for bugs)

1. Deploy multi-node calico cluster in IPIP mode with no outbound NAT
2. From a pod ping an external address (external to the cluster)
3. Return traffic is dropped on the FORWARD chain of whichever node is the next hop for the upstream BGP router

Your Environment

• Calico version: v3.26.1
• Orchestrator version (e.g. kubernetes, mesos, rkt): k3s
• Operating System and version: Debian 12, Linux k3s-control 6.1.0-10-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.38-2 (2023-07-27) x86_64 GNU/Linux

[mazdakn]

What's the value of keepOriginalNextHop in the BGPPeer? If set to true, then the return traffic should not be sent to another node.