search

metallb / metallb

A network load-balancer implementation for Kubernetes using standard routing protocols
https://metallb.universe.tf
Apache License 2.0
6.97k stars 903 forks source link

IP Pool Advertise/Broadcast Causing Network Security Policy to Block Port #1239

Closed beckyjmcdabq closed 2 years ago

beckyjmcdabq commented 2 years ago

We tried deploying MetalLB as our Kubernetes Baremetal load balancer. Everything appears to work for a short period of time and then the cluster networking just completely stops working.

We have determined that the issue is caused by: 1) MetalLB is advertising/broadcasting the pool of IP addresses (needed to route the traffic) 2) The network security policy/networks witch sees this broadcast coming from a specific MAC address and knows that the IP addresses (in the broadcast pool) do not belong to that MAC address 3) The network security policy then blocks all network traffic to/from that port

We have very little control over the network security policies.

Has anyone else encountered this issue? Is there a work around?

We are using BGP.

fedepaol commented 2 years ago

Are you sure you are using BGP? I am asking because you are referring to broadcast and mac addresses, which is exactly the way L2 work, with ARP broadcasts advertising an ip for a mac that does not belong to that ip. BGP is about routes and next hops, macs are not involved.

beckyjmcdabq commented 2 years ago

Actually I was informed that we are not using BGP. We configured Layer2.

rata commented 2 years ago

@beckyjmcdabq and are you REALLY sure that item 2 and 3 that you mentioned are the cause of the issue, and not something else? Can't that configuration be changed? Otherwise, if your router does that, it will never work. With metallb or probably any other alternative either in layer 2 mode.

If the router knows that one MAC has only one IP and blocks when it answers ARPs for others, then it will be impossible for one machine to advertise more than one IP. Therefore, it is not possible to work. But what we are doing here is done by tons of things, so it seems quite surprising that that is the issue

Assuming that is the case, you either change that config, the router or use BGP. A service won't be able to expose different IPs on that setup.

Are you really sure that is what is happening, though? I have never heard of such a thing in routers. I'm suspicious, and I guess other things could be happening (like gratious ARP confusing your router, etc.)

fedepaol commented 2 years ago

There could be anti mac-spoofing mechanisms in place, we hit them already (see for example https://github.com/metallb/metallb/issues/1147 , where the failover wasn't working more or less for the same reason). If that's the case, there is not much we can do as broadcasting ips for macs we don't own is the base of how L2 works.

fedepaol commented 2 years ago

I am gonna close this, feel free to reopen if things are not clear, but the arp mechanism is the base of L2 mode and we can't change that.