pfSense seems to be blocking connections

[johncarterofmars]

Describe the problem I have a private LAN, 192.168.200.0/24. I have 2 devices on this, both Linux. One is just a peer, while the other, an Ubuntu 20.04 VM, is a routing host. These are both behind a pfSense firewall. Both of these can connect to the controller at a self-hosted VM on AWS. Obviously, these two devices can connect to each other as they are on the same LAN. If I do netbird status (full details below), it shows that the desktop is connected to the routing node (keeper). However, on the laptop at another location, any location that is behind a pfsense firewall, the laptop will connect to the AWS hosted controller, but it is unable to connect to any other peer. Laptop cannot ping either of the other two (as it is not connected) and these 2 cannot ping the laptop. I get this message: Destination Host Unreachable ping: sendmsg: Required key not available I am sure this has something to do with how pfsense handles the returning wireguard traffic. I am not asking to help troubleshoot the pfsense part, its more about what direction to even go to do it. Is there a way to set static ports so I can configure the NAT in the firewall to those static ports? Is anyone else having issues with NAT and connections between other sites that are behind firewalls?

To Reproduce Add a peer behind a pfsense firewall Add another peer behind a pfsense firewall on another network Ping to each other

Expected behavior to be able to have devices behind different firewalls able to communicate with each other. I am looking for suggestions to help me figure out what the issue may be so I can resolve it.

NetBird status -d output:

sudo netbird status -d Peers detail: laptop.netbird.selfhosted: NetBird IP: 100.114.155.201 Public key: ppCYTEvMqrAIo6sCrj9euH7t1zTnERL4uCFX2Bh6sU0= Status: Connecting -- detail -- Connection type: P2P Direct: false ICE candidate (Local/Remote): srflx/host Last connection update: 2024-01-01 09:41:25

keeper.netbird.selfhosted: NetBird IP: 100.114.255.6 Public key: rit7uH0pG4kQb0owERzvoLwq84gzAk/dr354fSLV+QU= Status: Connected -- detail -- Connection type: P2P Direct: true ICE candidate (Local/Remote): srflx/host Last connection update: 2024-01-01 09:29:08

Daemon version: 0.25.2 CLI version: 0.25.2 Management: Connected to https://REDACTED_URL Signal: Connected to https://REDACTED_UR FQDN: desktop.netbird.selfhosted NetBird IP: 100.112.196.218/16 Interface type: Kernel Peers count: 1/2 Connected

Screenshots

Additional context

[johncarterofmars]

Does anyone have any suggestions on what I need to ensure is opened up on either incoming or outgoing traffic? It seems the NATing is causing the issue.

[Vignesh1230]

I've also had the same issue as im installing Netbird for the first time. My setup: EC2 Instance running Netbird self hosted. Local network pfSense firewall. A CentOS VM as a routing host on local network. Try connect using Cellular on iphone, but 0 of 2 peers connected.

user@tailscale-vpn:~$ sudo netbird status -d Peers detail: iphone.netbird.selfhosted: NetBird IP: 100.115.74.205 Public key: yPh2oHrVVYesmY4R0KxCLz3at2kmyMJdCDNlHALh02A= Status: Disconnected -- detail -- Connection type: Direct: false ICE candidate (Local/Remote): -/- Last connection update: 2024-01-08 14:37:01

mbp.netbird.selfhosted: NetBird IP: 100.115.117.141 Public key: 4VvAVFSblYeSFMKDblMkK2Yook7hQAV5gGM+TLsz4VQ= Status: Disconnected -- detail -- Connection type: Direct: false ICE candidate (Local/Remote): -/- Last connection update: 2024-01-08 14:28:46

Daemon version: 0.25.3 CLI version: 0.25.3 Management: Connected to https://REDACTED_URL Signal: Connected to https://REDACTED_URL FQDN: tailscale-vm.netbird.selfhosted NetBird IP: 100.115.51.88/16 Interface type: Kernel Peers count: 0/2 Connected

user@tailscale-vm~$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 82:c5:44:47:ff:6c brd ff:ff:ff:ff:ff:ff inet 10.0.1.82/20 brd 10.0.15.255 scope global dynamic ens18 valid_lft 4888sec preferred_lft 4888sec inet6 fe80::80c5:44ff:fe47:ff6c/64 scope link valid_lft forever preferred_lft forever 3: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500 link/none inet 100.69.15.30/32 scope global tailscale0 valid_lft forever preferred_lft forever inet6 fd7a:115c:a1e0::2b85:f1e/128 scope global valid_lft forever preferred_lft forever inet6 fe80::91fd:edc4:c7d5:ea25/64 scope link stable-privacy valid_lft forever preferred_lft forever 6: netmaker: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000 link/none inet 10.20.30.2/24 brd 10.20.30.255 scope global netmaker valid_lft forever preferred_lft forever 9: wt0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1280 qdisc noqueue state UNKNOWN group default qlen 1000 link/none inet 100.115.51.88/16 brd 100.115.255.255 scope global wt0 valid_lft forever preferred_lft forever

user@tailscale-vm:~$ ping 100.115.51.88 PING 100.115.51.88 (100.115.51.88) 56(84) bytes of data. ^C --- 100.115.51.88 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 3053ms

Im running Tailscale flawlessly right now (with the tailscale VM as the subnet gateway).

Nothing is working..

[johncarterofmars]

I am surprised there hasn't been any response. There is some traffic that must be being filtered to prevent this from working. I found this, but it didn't help me to solve the issue yet - https://forum.netgate.com/topic/175395/routing-from-subnet-does-not-belong-to-pfsense/19

[stevo11811]

Hey there, Please try outbound Static NAT, this did the trick for me BUT this can cause issues if you are using an exit node since the ports will not be randomized outbound. This requires careful planning.

Obviously replace the subnet with whatever you are using.

[vasquezmi]

Have you been able to solve for this? Also, what are you using as the proxy manager? Been reading that gRPC needs to be configured at the reverse-proxy to route traffic between the services and client.

[johncarterofmars]

No, I added the rule in NAT mentioned above but it didnt make a change.

[SamB-GB]

Same here I tried adding the NAT outbound but that didn't help.

I have a similar rule for Tailscale and it works:

https://tailscale.com/kb/1146/pfsense