BGP source IP address unpredictable when using CARP

[NielsPiersma]

Important notices Before you add a new report, we ask you kindly to acknowledge the following:

• [X] I have read the contributing guide lines at https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md
• [X] I have searched the existing issues, open and closed, and I'm convinced that mine is new.
• [X] When the request is meant for an existing plugin, I've added its name to the title.

Is your feature request related to a problem? Please describe. Yeah, It might be, root issue is that we "thought" when using CARP the lowest IP address is used for BGP packet exchange. However, this seems not to be the case. Sometimes the CARP IP address is used, and sometimes the REAL IP address is used. This leads to unpredictable results. At this moment, we add both IP addresses as neighbors. But I don't think this is a good and reliable solution. Adding the source IP address to the GUI would resolve this issue. We already can select the source interface, but when selecting a CARP interface, it has a minimum of TWO addresses. And it is a kinda roulette in which of the two IP addresses is used.

Describe the solution you'd like When setting selecting an interface, use always the lowest/highest address. Add the possibility for selecting Source Interface (already there) AND Source IP Address

Describe alternatives you've considered Added both IP addresses to BGP, but we have not regression-tested the stability of this setup. Understand, we need four boxes for testing this properly.

Additional context

Niels

[AdSchellevis]

If you want to force an address, it might be best to choose one using an outbound nat rule. Pushing these to the service in question can get flaky at some point in my experience.

[NielsPiersma]

If you want to force an address, it might be best to choose one using an outbound nat rule. Pushing these to the service in question can get flaky at some point in my experience.

I don't understand why I should create a NAT, glued to ONE of my CARP addresses for (outbound) traffic when FFR-BGP supports configuring an IP address. I can't oversee the impact of what will happen during a fail-over. I assume the NAT rule for the primary interface is propagated during SYNC and won't match the secondary.

Where / when did you experience flaky behavior when assigning/pushing an IP address to a service? For instance, when configuring DHCP HA fail-over, I need to configure the specific fail-over IP address for this to work.

Niels

[AdSchellevis]

I don't understand why I should create a NAT, glued to ONE of my CARP addresses for (outbound) traffic when FFR-BGP supports configuring an IP address. I can't oversee the impact of what will happen during a fail-over. I assume the NAT rule for the primary interface is propagated during SYNC and won't match the secondary.

Because it's a generic use case and works for all similar problems. In my honest opinion a lot of people ask for interface/address selections for various services for the wrong reasons, 99% of the cases can be solved with routing and address selection using pf. The reason why most services do support it is they usually are configured in static environments in which case the operator handles change manually anyway.

Where / when did you experience flaky behavior when assigning/pushing an IP address to a service? For instance, when configuring DHCP HA fail-over, I need to configure the specific fail-over IP address for this to work.

Changing the virtual ip address and people expecting the service to cope with it (which it wouldn't as it can not know anything about the change, but when organised with pf rules, there's only one facility to fix which already knows about changes)

[zagi-tng]

ISP/ bgp peer expects bgp session to be started from specific IP* address (security). *If CARP VIP mode is in use one would assume that that source peer IP should be VIP address.**

frr does check if he is CARP master: start_precmd="ifconfig | grep 'carp: MASTER'"

Problem is that it uses interface IP* address and not VIP one. And you can't 'NAT' BGP session info (router id and such). This option IS valid if used via vtysh and it works, but the os-frr plugin does not know about this command (update-source) and even deletes it from cfg if you enter it via shell/cli.

So. i am on shell cfg waiting for os-frr update. O:-/

[AdSchellevis]

@zagi-tng you actually tried changing the outbound address as suggested? If the address is part of the communication (which is the case with some protocols), it might not be possible, so far I only heard people asking for feature without context other then "the service supports this", which is obviously true, but in 90% of the cases a generic approach is more flexible and less sensitive to change.

Don't get me wrong, this isn't my plugin (feel free to open any PR and let the owner validate it), but for various services we receive similar questions quite often, which usually leads to the conclusion that it's not needed for obvious reasons (routing, nat usage, ..). Forcing addresses in services often leads to disappointments further down the road, causing support tickets, false expectations, etc etc.

That you can setup a feature doesn't always mean you should in the context of the rest of the ecosystem. In quite some cases you end up with a frankenstein monster.

[zagi-tng]

Yes. i did because i had no other choice. My uplink (bgp peer) expects bgp session to be from/to exact IP* address. BGP session are usualy in same ethernet prefix so no routing is done to get to other peer (ok.there is multihop BGP, but that is another story). If i leave it to os-frr, then the IP from WAN iface is used and session is rejected on other side. And ISP do not want cfg that add another and another IP peer to their setup. So i use update-source and when/if CARP finds that MASTER is down, BACKUP takes over and VIP source is used on backup node/frr . If os-frr could have an option to use vip ip as update-source "automagicaly" ? :-)

[NielsPiersma]

Hi all, Sorry for the delayed response; I've looked into this issue more closely and concluded that Ad's guidance regarding where the IP source addresses, in general, should be managed is a valid one.

It is impractical that OPNSense knows about every plug-in's specific requirement for IP addressing requirements. When looking at OPNSense as a firewall platform, the only location where you should mingle with IP addresses (vs. a purely routing platform) is via address translation.

So when configuring a neighbor who should be accessed from a specific source IP address at this machine, a source nat rule should work just fine; at least, it did during our testing for IPv4.

The trick is to configure a manual source nat rule (Firewall: NAT: Outbound, using manual or hybrid mode), selecting the traffic and choosing where it should originate from, for example, if a neighbor is residing at 10.10.10.10, but an outgoing connection to this neighbor should use (IP Alias, CARP, etc.) 10.10.10.2 as a source where 10.10.10.1 is also configured on the same interface, as the primary interface configured address, match the traffic using the following settings:

Destination address : 10.10.10.10
Translation / target: 10.10.10.2 

A more fine-grained approach is possible, like only translating traffic using port 179 (BGP in my specific case). For IPv6, this should operate similarly, and if it isn't possible to enforce an address, I recommend opening a ticket in core explaining which translation doesn't work.

The main advantage of using this approach is it functions for almost all services available, except those where the address is part of the conversation (for instance, SIP).

So even though I understand that when a plug-in supports custom IP addresses, it is, from OPNSense being a firewall platform, justifiable to say: "address mingling is configured via address translation," which works exactly the same for ALL plug-ins installed. As such, it is not a task for the plug-in but a task for the firewall's address translation logic.

Niels

[NiceRath]

Greetings.

As I'm not that deep into IPv6: if one uses IPv6 this should be doable with the NPTv6?

[zagi-tng]

For IPv6, this should operate similarly, and if it isn't possible to enforce an address, I recommend opening a ticket in core explaining which translation doesn't work.

It's IPv6. There is no NAT in IPv6.

What would be nice is if one could parse our VIP address from HA setup and use that as source IP.v6

[zagi-tng]

and there are things like Router-ID (can we NAT router-ID? O:-) )

http://networkerinterview.net/entries/eigrp-ospf-rip-notes/router-id-in-eigrp-ospf-bgp

[NielsPiersma]

@zagi-tng , I am not concerned about the router ID; that is 'merely' for identification and should be unique, usually, it is the lowest IP address on the box, and it is configurable via advanced under BGP settings.

Niels

[zagi-tng]

@zagi-tng , I am not concerned about the router ID; that is 'merely' for identification and should be unique

Usually it is not used since in 99% cases the length of AS-PATH is used.

the lowest IP address on the box, and it is configurable via advanced under BGP settings.

And it's also the IPv6 that i want to use. My CARP VIP IPv6 addresses are (higher) then my BGP update-source ipv6_address
I guess that is also the usual setup.

[bimbar]

FWIW I don't think SNAT on a BGP session is a particularly good solution.

Implementing

router bgp 64555
 neighbor foo update-source 192.168.0.1
 neighbor bar update-source lo0

for frr seems much saner.

[mimugmail]

Happy to review a pull request with those changes :)

[NielsPiersma]

FWIW I don't think SNAT on a BGP session is a particularly good solution.

Implementing

router bgp 64555
 neighbor foo update-source 192.168.0.1
 neighbor bar update-source lo0

for frr seems much saner.

@bimbar , Am I correct in assuming when using "neighbor bar update-source lo0" the kernel route table is referenced and as such all existing IP addresses in the kernel table are valid update sources?

Niels

[zagi-tng]

neighbor bar update-source lo0 @bimbar , Am I correct in assuming when using "neighbor bar update-source lo0" the kernel route table is referenced and as such all existing IP addresses in the kernel table are valid update sources?

i assume this won't work. The IPv4 and IPv6 (!) addresses bound to lo0 are not those the peer on other side expects. If i don't use update-source w/ exact IPv6 address bgp session is not enabled on other side.

[bimbar]

The point is, opnsense already can do neighbor foo update-source <interface>, probably the change should be to change the interface dropdown to an ip address field or ip address dropdown. But then I just came here from the WG-carp PR and don't even have an opnsense dev environment right now.

[AdSchellevis]

Anything that more or less suggests it can cope with change (like a dropdown showing the currently configured addresses), shouldn't be accepted for this (or any) plugin. I don't mind if anyone wants to push a static ip, nobody just has been able to explain why one would prefer that over a nat rule, which is generic, works for all services and is able to cope with change.

Setting a static address from the OS offers the same constraints as the translated one.

[mimugmail]

If this would be a closed source product and the software would set the nat automatically in the background like a just works solution, noone would have any concern :)

[AdSchellevis]

hehe, right, none of them is likely restarting services constantly when addresses change somewhere :)

[zagi-tng]

been able to explain why one would prefer that over a nat rule, which is generic,

IPv6

It has no NAT. Does not have to. On typical p2p link you get /64 CIDR. One of them is used for VIP - and that is what i want to use to start a bgp session towards my peer (isp) to whom i propagate my /48 prefix .

[AdSchellevis]

IPv6 ... It has no NAT. Does not have to. yes it does, maybe used for the exact same reasons, you just don't have to use private addresses internally.

I'm bailing out of this one, just mentioning design decisions should be made on proper grounds.

[OPNsense-bot]

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository, please read https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue, just let us know, so we can reopen the issue and assign an owner to it.