search

opnsense / core

OPNsense GUI, API and systems backend
https://opnsense.org/
BSD 2-Clause "Simplified" License
3.36k stars 753 forks source link

IPSec site-to-site is not connecting #5456

Closed kabaga closed 2 years ago

kabaga commented 2 years ago

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

Describe the bug

I am using WireGuard as my main site-to-site VPN tunnel between my OPNsense firewalls. I am trying to setup an route based IPSec site-to-site VPN as a secondary tunnel just in case the WireGuard goes down. I am using BGP to exchange routes and the IPSec seems to supports it.

The problem that I have is after configuring the IPSec, I could not reach the other end of the tunnel. I am getting "establishing IKE_SA failed, peer not responding". I am not using NAT traversal but it seems like the other OPNsense using it despite that it is disabled. 10[NET] <con1|1> sending packet: from MAIN-IP[4500] to REMOTE-IP[4500] (464 bytes). I think this is a problem. The remote site does not generating this NAT traversal log just from the main site.

Tip: to validate your setup was working with the previous version, use opnsense-revert (https://docs.opnsense.org/manual/opnsense_tools.html#opnsense-revert)

To Reproduce

Steps to reproduce the behavior:

  1. Configure both OPNsense firewalls as stated here https://docs.opnsense.org/manual/how-tos/ipsec-s2s-route.html
  2. Configure BGP for peering
  3. Ping the other end of the tunnel for response

Expected behavior

The IPsec s2s should come up and not use NAT traversal. The BGP peering should get established. The main OPNsense should be able to ping the VTI interface of the remote site.

Describe alternatives you considered

N/A

Screenshots Here is the settings. The remote site has mirrored the IP address. https://imgur.com/a/aIE8oz0

Relevant log files

https://imgur.com/a/a6IFQju

Additional context

N/A

Environment

Software version used and hardware type if relevant, e.g.:

Both devices are in version:

OPNsense 21.7.7-amd64 FreeBSD 12.1-RELEASE-p21-HBSD OpenSSL 1.1.1l 24 Aug 2021

The hardware: Main site: Supermicro A1SAi C2758 SoC 8GB RAM

Remote site: Protecli FW4B 8GB RAM

mimugmail commented 2 years ago

First of all, disable BGP and try to get up the tunnel. Screenshots would be fine. Also please be sure to not announce networks used for PtP links in VTI setup.

kabaga commented 2 years ago

I disabled the BGP neighbor config and no changes. Is the BGP forcing IPSec to use the NAT traversal?

I fixed the screenshot links

mimugmail commented 2 years ago

In you Phase1 you DISABLE nat traversal and in logs your endpoint wants/needs it. You should set it to enable or force as there is a nat device in between.

kabaga commented 2 years ago

There is no NAT device in between. Both OPNsense firewalls have public IP address. Also, the log screenshot the 4500/udp is coming from the main firewall and its config shows in the config screenshots.

I added inbound dst 4500/udp on WAN. I enabled the NAT traversal on both firewalls and I still can't reach the other end of the tunnel.

mimugmail commented 2 years ago

As I said, you need to set nat in Phase1 to force and restart

kabaga commented 2 years ago

I have enabled and forced NAT travelsal.

This is the logs I was getting

2022-01-05T10:04:48 charon[73003]   08[MGR] <con1|1> checkin of IKE_SA successful    
2022-01-05T10:04:48 charon[73003]   08[MGR] <con1|1> checkin IKE_SA con1[1]  
2022-01-05T10:04:48 charon[73003]   08[KNL] <con1|1> SADB_X_EXT_NAT_T_DPORT  
2022-01-05T10:04:48 charon[73003]   08[KNL] <con1|1> SADB_X_EXT_NAT_T_SPORT  
2022-01-05T10:04:48 charon[73003]   08[KNL] <con1|1> SADB_X_EXT_NAT_T_TYPE   
2022-01-05T10:04:48 charon[73003]   08[KNL] <con1|1> SADB_EXT_KEY_ENCRYPT    
2022-01-05T10:04:48 charon[73003]   08[KNL] <con1|1> SADB_EXT_KEY_AUTH   
2022-01-05T10:04:48 charon[73003]   08[KNL] <con1|1> SADB_EXT_ADDRESS_DST    
2022-01-05T10:04:48 charon[73003]   08[KNL] <con1|1> SADB_EXT_ADDRESS_SRC    
2022-01-05T10:04:48 charon[73003]   08[KNL] <con1|1> SADB_EXT_LIFETIME_CURRENT   
2022-01-05T10:04:48 charon[73003]   08[KNL] <con1|1> SADB_EXT_LIFETIME_SOFT  
2022-01-05T10:04:48 charon[73003]   08[KNL] <con1|1> SADB_EXT_LIFETIME_HARD  
2022-01-05T10:04:48 charon[73003]   08[KNL] <con1|1> SADB_X_EXT_SA2  
2022-01-05T10:04:48 charon[73003]   08[KNL] <con1|1> SADB_EXT_SA     
2022-01-05T10:04:48 charon[73003]   08[KNL] <con1|1> querying SAD entry with SPI c3f17662    
2022-01-05T10:04:48 charon[73003]   08[KNL] <con1|1> querying policy 0.0.0.0/0 === 0.0.0.0/0 in failed, not found    
2022-01-05T10:04:48 charon[73003]   08[KNL] <con1|1> querying policy 0.0.0.0/0 === 0.0.0.0/0 in  
2022-01-05T10:04:48 charon[73003]   08[MGR] IKE_SA con1[1] successfully checked out  
2022-01-05T10:04:48 charon[73003]   08[MGR] checkout IKEv2 SA with SPIs 05313e8772f4e430_i 2d250c8ebb1319cc_r    
2022-01-05T10:04:48 charon[73003]   08[MGR] <con1|1> checkin of IKE_SA successful    
2022-01-05T10:04:48 charon[73003]   04[NET] sending packet: from MAIN-SITE-PUBLIC-IP[4500] to REMOTE-SITE-PUBLIC-IP[4500]    
2022-01-05T10:04:48 charon[73003]   08[MGR] <con1|1> checkin IKE_SA con1[1]  
2022-01-05T10:04:48 charon[73003]   08[NET] <con1|1> sending packet: from MAIN-SITE-PUBLIC-IP[4500] to REMOTE-SITE-PUBLIC-IP[4500] (96 bytes)    
2022-01-05T10:04:48 charon[73003]   08[ENC] <con1|1> generating INFORMATIONAL response 17 [ ]    
2022-01-05T10:04:48 charon[73003]   08[ENC] <con1|1> parsed INFORMATIONAL request 17 [ ]     
2022-01-05T10:04:48 charon[73003]   08[NET] <con1|1> received packet: from REMOTE-SITE-PUBLIC-IP[4500] to MAIN-SITE-PUBLIC-IP[4500] (96 bytes)   
2022-01-05T10:04:48 charon[73003]   08[MGR] IKE_SA con1[1] successfully checked out  
2022-01-05T10:04:48 charon[73003]   08[MGR] checkout IKEv2 SA by message with SPIs 05313e87672f4e430_i 2d2590c8ebb1319cc_r   
2022-01-05T10:04:48 charon[73003]   03[NET] waiting for data on sockets  
2022-01-05T10:04:48 charon[73003]   03[NET] received packet: from REMOTE-SITE-PUBLIC-IP[4500] to MAIN-SITE-PUBLIC-IP[4500]   
2022-01-05T10:04:48 charon[73003]   03[NET] 96: 1E 9C B1 B5 ....     
2022-01-05T10:04:48 charon[73003]   03[NET] 80: 0D A6 9C DA 0A D1 74 79 FA 9D 20 B3 9D 51 B0 6F ......ty.. ..Q.o     
2022-01-05T10:04:48 charon[73003]   03[NET] 64: 6C 9A 88 71 68 BD 91 0B C1 56 61 9F AD 8B 79 25 l..qh....Va...y%     
2022-01-05T10:04:48 charon[73003]   03[NET] 48: 3A 7E F2 BB CD 2D 9C 2E 2E F7 41 54 77 26 45 DF :~...-....ATw&E.     
2022-01-05T10:04:48 charon[73003]   03[NET] 32: 00 00 00 44 6B 0B 04 BC 44 AE FF 3F D9 13 8E 2F ...Dk...D..?.../     
2022-01-05T10:04:48 charon[73003]   03[NET] 16: BB 13 19 CC 2E 20 25 00 00 00 00 11 00 00 00 60 ..... %........`     
2022-01-05T10:04:48 charon[73003]   03[NET] 0: 00 00 00 00 05 31 3E 87 72 F4 E4 30 2D 25 0C 8E .....1>.r..0-%..  
2022-01-05T10:04:48 charon[73003]   03[NET] received packet => 100 bytes @ 0x00006983c86c25e0
8191 commented 2 years ago

I am not using NAT traversal but it seems like the other OPNsense using it despite that it is disabled.

As you are using IKEv2, this parameter is ignored and negotiated. If there is any NAT performed in between, it's enabled, otherwise disabled. According to the log above in your case it is enabled, presumably as either end detects a NAT being in place or due to use of MOBIKE (which can be disabled in the OPNsense GUI).

Which connection mode do you have configured? default or Start immediate? Do you have the possibility to do a tcpdump on both ends?

From the strongSwan documentation:

nat_traversal = yes | no

activates NAT traversal by accepting source ISAKMP ports different from udp/500 and being able of floating to udp/4500 if a NAT situation is detected. Used by IKEv1 only, NAT traversal is always being active in IKEv2.

8191 commented 2 years ago

I've just checked the code and the OPNsense GUI setting NAT Traversal on/off in fact only controls the automatic generated firewall rules (add rule for udp/4500). Only the setting "force" has an impact to IKE itself.

So if you select NAT Traversal "off", but on IKE level actually NAT is negotiated (as in your case according to the logs), then the encaps traffic is not permitted by default. So, either set "NAT Traversal" to on, or create a custom firewall rule allowing udp/4500.

kabaga commented 2 years ago

i tried to set nat-t to enable and force, but the firewalls still could not ping each other. Can you share a screenshot that is known working route based IPsec site-to-site? If you can also provide a working the FRR BGP config with it that would be nice

8191 commented 2 years ago

p1 p2 gw route

kabaga commented 2 years ago

Is the local address needs to be the subnet? I didn't change any IP addresses and I am using the WAN interface otherwise my config is the same. However, it is still not working for me. I didn't create a static route since I will be using BGP, but regardless, I should be able to ping the tunnel interface

mimugmail commented 2 years ago

Can you Show a screenshot of IPsec Status page, Routing table and check packets in enc0

8191 commented 2 years ago

Is the local address needs to be the subnet?

No. The local address is any (virtual) transit IP. Does not even need to be part of a valid net (not even a netmask is provided). I use e.g. 172.16.0.0 for one side and 172.16.0.8 for the other side. The IP is later assigned to the virtual ipsec* interface and therefore should not be part of any other physical or logical network interface on any of your nodes.

kabaga commented 2 years ago

Can you Show a screenshot of IPsec Status page, Routing table and check packets in enc0

Here is the screenshots of the status overview page of both firewalls. On the remote site, there is no entry for 192.168.1.1 which is the IP address of the Main site's IPsec interfaces.

https://imgur.com/a/GjeGRPP

OPNsense-bot commented 2 years ago

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository, please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue, just let us know, so we can reopen the issue and assign an owner to it.