search

projectcalico / calico

Cloud native networking and network security
https://docs.tigera.io/calico/latest/about/
Apache License 2.0
5.94k stars 1.32k forks source link

Unable to access BGP advertised service LB IPs from host itself #9152

Closed veebkolm closed 1 month ago

veebkolm commented 1 month ago

I cannot access Service LoadBalancer IPs from the node itself where the IP is advertised from.

Expected Behavior

I should be able to access BGP advertised Service LB IPs from the node itself.

Current Behavior

I cannot access Service LoadBalancer IPs from the node itself where the IP is advertised from. Curl doesn't work except when using curl --interface eth0 - then it does.

Possible Solution

I understand why blackhole is necessary. I also can't find a way to make CRI-O use eth0 like I can make curl use it.

Steps to Reproduce (for bugs)

  1. Install Calico and configure BGP peering - both pods and Service LB IPs
    • BGPPeer has peerIP, asNumber, password and keepOriginalNextHop: true
    • BGPConfiguration has serviceLoadBalancerIPs
  2. Install MetalLB without speakers but with IP pools
  3. Create a service that will get a LB IP assigned
  4. External and in-pod connectivity to that IP works but not from the host system

Context

The LB IP serves Traefik, which proxies Harbor, from where CRI-O needs to pull images from. The blackhole IP route seems to be the issue, when removing it, connectivity works for a brief moment (until Calico/bird re-adds it).

Your Environment

caseydavenport commented 1 month ago

@veebkolm could you share your ip route show output from the node? My memory of this is that we shouldn't be programming blackhole routes into the kernel - I believe we should have an programming filter in the BIRD configuration to prevent actually programming these routes (and instead only advertise them over BGP).

https://github.com/projectcalico/calico/blob/0c68b6a4204eeaf77b9a832a86c3e1a56d546520/confd/etc/calico/confd/templates/bird_ipam.cfg.template#L81-L86

veebkolm commented 1 month ago

@caseydavenport indeed the mentioned IP is blackholed on the node it's scheduled on

default via 172.21.255.254 dev eth0 proto static 
10.244.61.0/26 via 172.21.3.62 dev eth0 proto 80 onlink 
10.244.73.192/26 via 172.21.3.53 dev eth0 proto 80 onlink 
10.244.86.192/26 via 172.21.3.89 dev eth0 proto 80 onlink 
10.244.87.0/26 via 172.21.3.63 dev eth0 proto 80 onlink 
10.244.114.128/26 via 172.21.3.82 dev eth0 proto 80 onlink 
10.244.117.0/26 via 172.21.3.61 dev eth0 proto 80 onlink 
10.244.138.128/26 via 172.21.3.72 dev eth0 proto 80 onlink 
10.244.144.64/26 via 172.21.3.52 dev eth0 proto 80 onlink 
10.244.149.192/26 via 172.21.3.83 dev eth0 proto 80 onlink 
10.244.150.0/26 via 172.21.3.83 dev eth0 proto 80 onlink 
10.244.151.64/26 via 172.21.3.74 dev eth0 proto 80 onlink 
10.244.153.0/26 via 172.21.3.71 dev eth0 proto 80 onlink 
blackhole 10.244.157.0/26 proto 80 
10.244.157.1 dev cali679c3b14d02 scope link 
10.244.157.3 dev calic8d9edafbb2 scope link 
10.244.157.4 dev calid08563a4014 scope link 
10.244.157.5 dev calia42e8bd9301 scope link 
10.244.157.9 dev cali7dc08daabaf scope link 
10.244.157.10 dev cali61f8fc2ce94 scope link 
10.244.157.11 dev cali68d73000cc8 scope link 
10.244.157.12 dev calid6a8645dc20 scope link 
10.244.157.13 dev cali19c8069d07c scope link 
10.244.157.16 dev cali6888104c6fc scope link 
10.244.157.18 dev cali19f5478e121 scope link 
10.244.157.21 dev calie974d7a215b scope link 
10.244.157.22 dev cali44abbaf50b4 scope link 
10.244.157.25 dev cali5e7c5aa1e98 scope link 
10.244.157.27 dev cali067c2c771cf scope link 
10.244.157.28 dev cali7df7d5f671b scope link 
10.244.157.29 dev cali4e54d016e94 scope link 
10.244.157.30 dev cali3631c1238a3 scope link 
10.244.157.31 dev cali027d9c24216 scope link 
10.244.157.32 dev cali4a6a08c964c scope link 
10.244.157.35 dev caliebd7186724c scope link 
10.244.157.36 dev calif88c704f105 scope link 
10.244.157.37 dev cali13f87f75701 scope link 
10.244.157.38 dev cali3b91bd13632 scope link 
10.244.157.39 dev cali79a163d84d2 scope link 
10.244.157.40 dev cali20656abe98f scope link 
10.244.157.41 dev cali0b275158e7a scope link 
10.244.157.42 dev cali24e15c13a03 scope link 
10.244.157.43 dev calib45740b2ad7 scope link 
10.244.157.44 dev cali38064ecbbe5 scope link 
10.244.157.46 dev cali7e5e498208a scope link 
10.244.157.49 dev cali411b5751668 scope link 
10.244.157.51 dev calidb975f0b501 scope link 
10.244.157.54 dev cali0deaa2bc0e2 scope link 
10.244.157.57 dev cali6d40b5976c5 scope link 
10.244.157.58 dev cali573f8369a31 scope link 
10.244.157.60 dev calie89d30894a1 scope link 
10.244.217.0/26 via 172.21.3.73 dev eth0 proto 80 onlink 
10.244.224.0/26 via 172.21.3.51 dev eth0 proto 80 onlink 
10.244.250.0/26 via 172.21.3.84 dev eth0 proto 80 onlink 
10.244.250.64/26 via 172.21.3.84 dev eth0 proto 80 onlink 
10.244.250.128/26 via 172.21.3.84 dev eth0 proto 80 onlink 
blackhole x.x.x.x proto bird
veebkolm commented 1 month ago

Got it working by changing following things:

  1. Updated from Ubuntu 22 to 24
  2. Set kube-proxy mode to "" not ipvs
  3. IPPool spec ended up being: 
      allowedUses:
    - Workload
    - Tunnel
  blockSize: 26
  cidr: 10.244.0.0/16
  ipipMode: Always
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: Never