search

theOtherLuke / nordlynx-router

Instructions for creating a NordVPN router on (ideally)any linux host using the nordlynx protocol. So far the success list includes Debian 12, Ubuntu 24.04, and Fedora 38-40.
MIT License
2 stars 0 forks source link

Routing issue #3

Open nielsnetsec opened 2 months ago

nielsnetsec commented 2 months ago

Hi,

First, thanks for your work it is really appreciated.

I have setup the VPN gateway for days now, continuous issues with reaching internet from Client when the VPN is active. When the VPN is not active the connection is fine. My own setup was nearly the same as yours, but after days troubleshooting it does not seem to work.

Today I spend on setting up your configurations, unfortunately no luck:

:~$ cat /proc/sys/net/ipv4/ip_forward 1

Iptables:

dasf34@dasf34:~$ cat /etc/iptables/rules.v4

Ofcourse I will tighten the firewall, but for now I leave it open for testing (frustration) purposes.

tcpdump shows pings to 1.1.1.1 going out of the Nordlynx interface.

:~$ sudo tcpdump -i nordlynx 19:27:21.002091 IP 103.86.96.100.domain > dasf34.46402: Flags [S.], seq 2180119783, ack 72615275, win 65535, options [mss 1 380,sackOK,TS val 4118153234 ecr 3693106227,nop,wscale 9], length 0 19:27:21.002106 IP 103.86.96.100.domain > 172.16.179.1.46402: Flags [S.], seq 2180119783, ack 72615275, win 65535, options [mss 1380,sackOK,TS val 4118153234 ecr 3693106227,nop,wscale 9], length 0 19:27:21.014227 IP 103.86.96.100.domain > dasf34.46402: Flags [S.], seq 2180119783, ack 72615275, win 65535, options [mss 1 380,sackOK,TS val 4118153246 ecr 3693106227,nop,wscale 9], length 0 19:27:21.014242 IP 103.86.96.100.domain > 172.16.179.1.46402: Flags [S.], seq 2180119783, ack 72615275, win 65535, options [mss 1380,sackOK,TS val 4118153246 ecr 3693106227,nop,wscale 9], length 0 19:27:21.165976 IP 172.16.179.1 > one.one.one.one: ICMP echo request, id 5, seq 301, length 64

Now what's interesting imo, it seems to send traffic to the internal network (172.x.x.x) back out of the WAN.

:~$ ping 172.16.179.1 PING 172.16.179.1 (172.16.179.1) 56(84) bytes of data. From 192.168.179.254: icmp_seq=2 Redirect Host(New nexthop: 192.168.179.107) ^C --- 172.16.179.1 ping statistics --- 8 packets transmitted, 0 received, 100% packet loss, time 7169ms

When I look at Iptables, forwarding should be correct:

:~$ sudo iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp spts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp dpt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp spt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp dpts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp spts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp dpt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp spt:22 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spt:ssh / nordvpn / ACCEPT all -- 172.16.179.0/30 anywhere / nordvpn / ACCEPT all -- 192.168.179.0/24 anywhere / nordvpn / ACCEPT all -- 172.16.179.0/30 anywhere / nordvpn / ACCEPT all -- 192.168.179.0/24 anywhere / nordvpn / ACCEPT all -- anywhere anywhere connmark match 0xe1f1 / nordvpn / ACCEPT all -- anywhere anywhere connmark match 0xe1f1 / nordvpn / DROP all -- anywhere anywhere / nordvpn / DROP all -- anywhere anywhere / nordvpn /

Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp spts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp dpt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp spt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp dpts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp spts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp dpt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp spt:22 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spt:ssh / nordvpn / ACCEPT all -- anywhere 172.16.179.0/30 / nordvpn / ACCEPT all -- anywhere 192.168.179.0/24 / nordvpn / ACCEPT all -- anywhere 172.16.179.0/30 / nordvpn / ACCEPT all -- anywhere 192.168.179.0/24 / nordvpn / CONNMARK all -- anywhere anywhere mark match 0xe1f1 / nordvpn / CONNMARK save ACCEPT all -- anywhere anywhere connmark match 0xe1f1 / nordvpn / CONNMARK all -- anywhere anywhere mark match 0xe1f1 / nordvpn / CONNMARK save ACCEPT all -- anywhere anywhere connmark match 0xe1f1 / nordvpn / DROP all -- anywhere anywhere / nordvpn / DROP all -- anywhere anywhere / nordvpn /

Last technical

Upon VPN connection I cannot reach the LAN interface on the Host from the Client, I do not see in iptables why this is happening. Input, Output for these subnets seem to be fine.

I feel there is a simple solution, my experience with iptables and linux is a bit limited though (better than most probably).

Can you help me out on this one?

Much thanks and kind regards, Niels

theOtherLuke commented 2 months ago

Sorry to hear about your troubles with this. I will look into this as I have a little time. Might not be until this weekend. I will follow up then.

theOtherLuke commented 2 months ago

Just a thought, Ubuntu uses netplan to manage network interfaces. Have you tried adapting the network settings to use a netplan config file? Netplan config files are located in /etc/netplan/ and are .yaml format. Editing /etc/network/interfaces will have no effect. I apologize for not accounting for this. I haven't used Ubuntu in a while and I forgot about this key difference.

After you edit a netplan config, ( lots of info on the google for editing these files )

sudo nano /etc/netplan/<config-file>.yaml

try the config file using:

sudo netplan try

and if it works, apply it by pressing [ENTER] before it times out.

UPDATE: Changing from netplan to ifupdown will not fix an underlying DNS issue. I am leaving this here to show how to change incase someone prefers one over the other.

Alternatively, you might try removing netplan and installing ifupdown, then the configuration in the steps should work.

sudo apt remove netplan* -y && sudo apt install ifupdown -y

I'll workshop a way to use netplan and post an update.

Let me know if this works.

hoemada commented 2 months ago

Hi ! i had also an issue on the client, i wasn't able to ping the gateway(any other servers on other vlans) or internet. My servers are mostly debian lxc. To resolve this issue : 1 - i found the DHCP was working, my client received an IP, but the client has no gateway, so the default route wasn't installed. I correct it on my "router-nordvpn", i add the bellow line in the dnsmasq,conf : 

     interface=eth1 #eth1 is my lan interface
     dhcp-range=192.168.0.2,192.168.0.20,255.255.255.0,12h
+  dhcp-option=3,192.168.0.1 #dhcp option 3 is to set the gateway
+  dhcp-option=6,192.168.1.254 #dhcp option 6 is my homelab pihole/dns (from an other vlan id)

2 - i had to set nordvpn with the option "lan-discovery", my nord vpn settings : nordvpn settings

Technology: NORDLYNX
Firewall: enabled
Firewall Mark: 0xe1f1
Routing: enabled
Analytics: disabled
Kill Switch: enabled
Threat Protection Lite: disabled
Notify: disabled
Tray: enabled
Auto-connect: enabled
IPv6: disabled
Meshnet: disabled
DNS: 192.168.1.254
LAN Discovery: enabled
Virtual Location: disabled
theOtherLuke commented 2 months ago

FWIW, I did a quick test. This is inconclusive, but I setup 2 lxc containers. One Debian 12, the other Ubuntu 23.10 with all other details being the same. Network interface set to the same as my VPN container's WAN.

/etc/network/interfaces in Debian

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

/etc/netplan/config.yaml in Ubuntu

network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: true

All other settings unchanged. Both use networkd on the backend. Out of the box, the Debian container doesn't seem to have the same DNS issues as Ubuntu. Simply setting the DNS Server under DNS->DNS Server in Proxmox fixed Ubuntu for me. Switching Ubuntu from netplan to ifupdown did not fix anything. I did not try routing through either container yet. Maybe tomorrow.

I appreciate all the input and help working the kinks out. Thank you.

nielsnetsec commented 2 months ago

Hi,

Continuing with Ubuntu Server 23:

Looking in TCP dump and the fact that (from Host) a ping to my Client (on LAN) seems to be forwarded to WAN, it seems that there is an routing issue. I do not see where this is coming from however, Forwards seem fine and the Routing tables also sees the LAN network on the correct interface.

Firewall has no rules that prevent traffic TO LAN, but this should not matter anyway since I see traffic TO LAN forwarded to WAN:

:~$ ping 172.16.179.1 PING 172.16.179.1 (172.16.179.1) 56(84) bytes of data. From 192.168.179.254: icmp_seq=2 Redirect Host(New nexthop: 192.168.179.107)

I have purged Netplan and installed IfUp, which works fine but does not change the behaviour upon VPN login:

I am now installing Debian and will follow your guide to replicate your config.

Looking forward to assist on making this work on Ubuntu too, much fun :)

nielsnetsec commented 2 months ago

Hi,

Informing you that on Debian the internet does not work on the client, after following your guide. I have firewall open for Input, Output and Forward but unfortunately this does not work.

Forwarding

:~$ cat /proc/sys/net/ipv4/ip_forward 1

Firewall

:~$ cat /etc/iptables/rules.v4

Firewall when connected to VPN

:~$ sudo iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp spts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp dpt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp spt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp dpts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp spts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp dpt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp spt:22 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spt:ssh / nordvpn / ACCEPT all -- anywhere anywhere connmark match 0xe1f1 / nordvpn / ACCEPT all -- anywhere anywhere connmark match 0xe1f1 / nordvpn / DROP all -- anywhere anywhere / nordvpn / DROP all -- anywhere anywhere / nordvpn /

Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp spts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp dpt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp spt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp dpts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp spts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp dpt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp spt:22 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spt:ssh / nordvpn / CONNMARK all -- anywhere anywhere mark match 0xe1f1 / nordvpn / CONNMARK save ACCEPT all -- anywhere anywhere connmark match 0xe1f1 / nordvpn / CONNMARK all -- anywhere anywhere mark match 0xe1f1 / nordvpn / CONNMARK save ACCEPT all -- anywhere anywhere connmark match 0xe1f1 / nordvpn / DROP all -- anywhere anywhere / nordvpn / DROP all -- anywhere anywhere / nordvpn /

NordVPN Version 3.18.3 :~$ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 12 (bookworm) Release: 12 Codename: bookworm

Please let me know if I can do anything else to troubleshoot or assist you on making this work. I'd be happy to help.

Kind regards and thanks for your efforts, Niels

hoemada commented 2 months ago

Hello, can you post the output of nordvpn settings ? Also the routing table from both client and host ?

theOtherLuke commented 2 months ago

These are 2 bare containers without nordvpn installed yet. I wanted to start with testing how they work out of the box. I am starting the process of setting up nordvpn tonight.

theOtherLuke commented 2 months ago

Initial testing seems to indicate Ubuntu is not a good candidate for this use case until someone figures out the DNS/gateway issues that seem to be unique to Ubuntu.

On Ubuntu I have issues when trying to install dnsmasq. Probably because systemd-resolved is being used for DNS resolution. This appears to conflict with dnsmasq. I'm sure there is a method to make this work with systemd-resolved or maybe some other DHCP server, but as I universally prefer Debian over Ubuntu it isn't a priority for me to figure out right now. Maybe I'll play with it though if for no other reason than completeness.

However, I still have no problem with this when I set it up from scratch using Debian. It simply just works for me. I recommend using Debian 12 for this for the time being. I will update the main readme.md to reflect this.

These are the outputs from my newest test install on a Debian lxc. Maybe they'll help someone.

Output of $ nordvpn settings

~# nordvpn settings
Technology: NORDLYNX
Firewall: enabled
Firewall Mark: 0xe1f1
Routing: enabled
Analytics: disabled
Kill Switch: disabled
Threat Protection Lite: enabled
Notify: disabled
Tray: disabled
Auto-connect: enabled
IPv6: disabled
Meshnet: disabled
DNS: disabled
LAN Discovery: disabled
Virtual Location: enabled

Contents of /etc/iptables/rules.v4

See correct iptables in the post further down. You need the mangle rule.

~# cat /etc/iptables/rules.v4 
# Generated by iptables-save v1.8.9 (nf_tables) on Sat Aug 17 02:14:24 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A FORWARD -i eth1 -o nordlynx -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth1 -o nordlynx -j ACCEPT
COMMIT
# Completed on Sat Aug 17 02:14:24 2024
# Generated by iptables-save v1.8.9 (nf_tables) on Sat Aug 17 02:14:24 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -o nordlynx -j MASQUERADE
COMMIT
# Completed on Sat Aug 17 02:14:24 2024

$ iptables -L when connected

~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             connmark match  0xe1f1 /* nordvpn */
DROP       all  --  anywhere             anywhere             /* nordvpn */

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
CONNMARK   all  --  anywhere             anywhere             mark match 0xe1f1 /* nordvpn */ CONNMARK save
ACCEPT     all  --  anywhere             anywhere             connmark match  0xe1f1 /* nordvpn */
DROP       all  --  anywhere             anywhere             /* nordvpn */

Contents of /etc/dnsmasq.conf

~# cat /etc/dnsmasq.conf
interface=eth1
dhcp-range=192.168.55.50,192.168.55.100,12h

Contents of /etc/network/interfaces

~# cat /etc/network/interfaces
# loopback
auto lo
iface lo inet loopback

# WAN
auto eth0
iface eth0 inet dhcp

# LAN
auto eth1
iface eth1 inet static
        address 192.168.55.1/24

None of my test clients have ever needed to have special routing tables or rules applied on them. I simply set as dhcp or set a static address in the correct subnet like I would on any other system and it works. One of my requirements is to have it plug and play for my clients.

If anyone knows how to implement a DHCP server with systemd-resolved, or in a systemd-resolved friendly way, I welcome the assitance.

theOtherLuke commented 2 months ago

I sat down and figured out the basic steps to make this work on ubuntu 23.10. I have posted a rough draft in the ubuntu folder of this project.

https://github.com/theOtherLuke/nordlynx-router/tree/main/ubuntu

Let me know if this helps.

theOtherLuke commented 2 months ago

Hi,

Informing you that on Debian the internet does not work on the client, after following your guide. I have firewall open for Input, Output and Forward but unfortunately this does not work.

@nielsnetsec is routing enabled in nordvpn settings? Routing is required to make this work. I did not have that detail in the write up before, but I have added it now.

nordvpn routing on

nielsnetsec commented 2 months ago

Hi,

Thanks for your responses and research. I dont really care about using Ubuntu, if Debian works fine then i'm good with that.

Using Debian, to answer some question you had, let me put the thing here:

NordVPN=Disconnected - Forwarding from Client works fine.

` Linux 6.1.0-23-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15) x86_64

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law.

:~$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default ql en 1000 link/ether 00:0c:29:b2:7d:0e brd ff:ff:ff:ff:ff:ff altname enp2s1 inet 192.168.179.104/24 brd 192.168.179.255 scope global dynamic ens33 valid_lft 171735sec preferred_lft 171735sec 3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default ql en 1000 link/ether 00:0c:29:b2:7d:18 brd ff:ff:ff:ff:ff:ff altname enp2s2 inet 172.16.179.2/30 brd 172.16.179.3 scope global ens34 valid_lft forever preferred_lft forever

:~$ ip route default via 192.168.179.254 dev ens33 172.16.179.0/30 dev ens34 proto kernel scope link src 172.16.179.2 192.168.179.0/24 dev ens33 proto kernel scope link src 192.168.179.104

:~$ nordvpn settings Technology: NORDLYNX Firewall: enabled Firewall Mark: 0xe1f1 Routing: enabled Analytics: enabled Kill Switch: disabled Threat Protection Lite: disabled Notify: disabled Tray: enabled Auto-connect: enabled IPv6: disabled Meshnet: disabled DNS: disabled LAN Discovery: disabled Virtual Location: enabled Allowlisted ports: 22 (UDP|TCP) 67 - 68 (UDP|TCP) Allowlisted subnets: 172.16.179.0/30 192.168.179.0/24

:~$ nordvpn set routing on Routing is already set to 'enabled'.

:~$ cat /etc/iptables/rules.v4

:~$ sudo iptables -L [sudo] password for dasf34: Chain INPUT (policy ACCEPT) target prot opt source destination

Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT) target prot opt source destination :~$ '

NordVPN is connected - Forwarding from client does not work, Local (Client) Gateway is not responding.

:~$ nordvpn c ro71 Connecting to Romania #71 (ro71.nordvpn.com) You are connected to Romania #71 (ro71.nordvpn.com)!

:~$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group 57841 qlen 1000 link/ether 00:0c:29:b2:7d:0e brd ff:ff:ff:ff:ff:ff altname enp2s1 inet 192.168.179.104/24 brd 192.168.179.255 scope global dynamic ens33 valid_lft 171565sec preferred_lft 171565sec 3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group 57841 qlen 1000 link/ether 00:0c:29:b2:7d:18 brd ff:ff:ff:ff:ff:ff altname enp2s2 inet 172.16.179.2/30 brd 172.16.179.3 scope global ens34 valid_lft forever preferred_lft forever 8: nordlynx: <POINTOPOINT,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000 link/none inet 10.5.0.2/32 scope global nordlynx valid_lft forever preferred_lft forever

:~$ ip route default via 192.168.179.254 dev ens33 172.16.179.0/30 dev ens34 proto kernel scope link src 172.16.179.2 192.168.179.0/24 dev ens33 proto kernel scope link src 192.168.179.104

:~$ nordvpn settings Technology: NORDLYNX Firewall: enabled Firewall Mark: 0xe1f1 Routing: enabled Analytics: enabled Kill Switch: disabled Threat Protection Lite: disabled Notify: disabled Tray: enabled Auto-connect: enabled IPv6: disabled Meshnet: disabled DNS: disabled LAN Discovery: disabled Virtual Location: enabled Allowlisted ports: 22 (UDP|TCP) 67 - 68 (UDP|TCP) Allowlisted subnets: 172.16.179.0/30 192.168.179.0/24

:~$ nordvpn set routing on Routing is already set to 'enabled'.

:~$ cat /etc/iptables/rules.v4

:~$ sudo iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp spts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp dpt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp spt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp dpts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp spts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp dpt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp spt:22 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spt:ssh / nordvpn / ACCEPT all -- 192.168.179.0/24 anywhere / nordvpn / ACCEPT all -- 172.16.179.0/30 anywhere / nordvpn / ACCEPT all -- 192.168.179.0/24 anywhere / nordvpn / ACCEPT all -- 172.16.179.0/30 anywhere / nordvpn / ACCEPT all -- anywhere anywhere connmark match 0xe1f1 / nordvpn / ACCEPT all -- anywhere anywhere connmark match 0xe1f1 / nordvpn / DROP all -- anywhere anywhere / nordvpn / DROP all -- anywhere anywhere / nordvpn /

Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp spts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp dpt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp spt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp dpts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp spts:bootps:bootpc / nordvpn / ACCEPT udp -- anywhere anywhere udp dpt:22 / nordvpn / ACCEPT udp -- anywhere anywhere udp spt:22 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spts:67:68 / nordvpn / ACCEPT tcp -- anywhere anywhere tcp dpt:ssh / nordvpn / ACCEPT tcp -- anywhere anywhere tcp spt:ssh / nordvpn / ACCEPT all -- anywhere 192.168.179.0/24 / nordvpn / ACCEPT all -- anywhere 172.16.179.0/30 / nordvpn / ACCEPT all -- anywhere 192.168.179.0/24 / nordvpn / ACCEPT all -- anywhere 172.16.179.0/30 / nordvpn / CONNMARK all -- anywhere anywhere mark match 0xe1f1 / nordvpn / CONNMARK save ACCEPT all -- anywhere anywhere connmark match 0xe1f1 / nordvpn / CONNMARK all -- anywhere anywhere mark match 0xe1f1 / nordvpn / CONNMARK save ACCEPT all -- anywhere anywhere connmark match 0xe1f1 / nordvpn / DROP all -- anywhere anywhere / nordvpn / DROP all -- anywhere anywhere / nordvpn /

Comments

As you can see, the firewall has changed massively and the local Client gateway (172.16.179.2) is not reachable. I think that the cause for the unreachable gateway can be found in the Redirected Next-Hop.

Ping Client from Debian: VPN-GW-DEBIAN:~$ ping 172.16.179.1 PING 172.16.179.1 (172.16.179.1) 56(84) bytes of data. From 192.168.179.254: icmp_seq=2 Redirect Host(New nexthop: 192.168.179.104) ^C --- 172.16.179.1 ping statistics --- 5 packets transmitted, 0 received, 100% packet loss, time 4048ms

The Client settings and routing are just basic, firewall is turned off. Nothing is needed here ofcourse:

:~$ ip route default via 172.16.179.2 dev ens33 proto dhcp metric 100 169.254.0.0/16 dev ens33 scope link metric 1000 172.,16.179.0/30 dev ens33 proto kernel scope link src 172.16.179.1 metric 100

My idea of cause:

I think that upon connecting with NordVPN the Client is routed through WAN and traffic never reaches the Client. Even the ICMP pings from Client to its gateway can be seen in tcpdump going through the WAN. I do not see where this is happening.

I am curious why NordVPN adds such a massive amount of Firewall upon connected, which it does not on your side. Here we can see a big difference between you and me. What triggers this? (EDIT) Eventhouh the Firewall is default set to Allow All, NordVPN has rules that block Any-Any traffect below. I tried to remove this but it does not seem to work. Also, I do not see any routing that send Client (return) traffic back to WAN. (EDIT)

I think it is plausible this change in firewall is what causes the Client Gateway being not reachable, because traffic is not forwarded back to the client. I cannot see where this is triggered however.

Am I missing something obvious here?

Thanks again for your efforts, kind regards,

theOtherLuke commented 2 months ago

I apologize @nielsnetsec. I posted the wrong iptables output. I'll update tonight. Nord does dynamically add a number of rules on connection.

I did notice you have both your networks whitelisted. That is not needed. When it's working correctly, you also should not need to whitelist the dns ports either.

That's just my initial thoughts. I'll look closer tonight when I get home.

theOtherLuke commented 2 months ago

@nielsnetsec Here is the correct iptables while connected. It's very minimal unless things start getting added to the whitelist. Basically, anything added to the whitelist, multiply by the number network interfaces. 2 interfaces x 2 whitelist items = 4 new rules. I suggest that unless you know for certain you absolutely need a particular item whitelisted, such as port 22 (ssh) if you are connecting through the WAN interface to configure, don't add it until you have the basic setup working. Read on and I'll explain why you basically don't need any whitelist items.

As for nordvpn adding rules to drop traffic, nordvpn has a rule that drops all traffic that doesn't possess the firewall marker. I did figure out how to mark the traffic, but then I realized that forwarded traffic never sees the input or output. The input and output tables only affect the localhost, or in this case, the nordvpn host. The path for forward traffic is:

--------------------------------------------------------------
| ->interface->prerouting->forward->postrouting->interface-> | <<< client path to/from internet
--------------------\-------------------/---------------------
                   input             output  <<< nordvpn rules are here
                      \-->localhost-->/

So, what I'm trying to say is this: since nordvpn only adds rules to the input and output chains which means nordvpn's rules don't affect forward traffic in any way unless something is configured really wrong. Nord's rules do serve to protect the machine it's installed on though.

Nord's rules are intended to apply only to the localhost because the app is intended to protect the machine it's installed on. That means any whitelist items only apply to, you guessed it, the localhost. Again, this is because the rules are only on tables that apply to localhost, not forward traffic. The only whitelist that should ever be needed, and only temporarily, is if you are configuring over the WAN interface you need to whitelist port 22 for ssh.

I also want to make sure we are talking about the same things when we say host or client etc. This is a "no stone unturned" approach, just in case.

host aka localhost - the machine/container/vm on which nordvpn is installed and through which traffic is passing to get to the internet or outside network. client - any machine, vm, container, or device that connects to the internet, or any outside network, through the host by way of the LAN interface. WAN - port on host that connects to the internet. LAN - port on the host that connects to the client network.

root@debian:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             connmark match  0xe1f1 /* nordvpn */
DROP       all  --  anywhere             anywhere             /* nordvpn */

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
CONNMARK   all  --  anywhere             anywhere             mark match 0xe1f1 /* nordvpn */ CONNMARK save
ACCEPT     all  --  anywhere             anywhere             connmark match  0xe1f1 /* nordvpn */
DROP       all  --  anywhere             anywhere             /* nordvpn */

Long story short, clean up your iptables rules, and if you're still having trouble, check the DNS settings on your WAN connection. This could be the hypervisor(Proxmox, VMWare, etc. or router upstream(most likely your modem). I don't have this issue with my Debian setups, but I do on Fedora and Ubuntu. The fix is to assign the DNS Server in the container settings on the hypervisor or in your dns/dhcp configuration on the host(nordvpn). The DNS Domain is the networks domain - example.com or mydomain.net etc. If your DNS and DHCP are different addresses from your nordvpn instance, it may be easier to configure in the host's dns/dhcp config file. @hoemada showed how to do this in a post earlier on this thread if you're using dnsmasq. kea-dhcp(isc-dhcp replacement from isc) has similar capabilities. Probabaly isc-dhcp too. However, isc-dhcp is end-of-life and switching to one of the others is recommended by isc.

I generally let the nordvpn container connect to my modem using DHCP. Then my pfsense connects to the nordvpn container by DHCP. I don't open any doors that don't need to be opened on the vpn and let pfsense do the heavy firewall lifting. My pfsense box then handles all DHCP itself and direction to my pihole for DNS for the clients on my private network. For outside access I have 1 extra path connecting pfsense directly to the modem, but pfsense is configured to only allow traffic on 1 specific port to 1 specific target running containerized wireguard. This way, if for some reason my nordvpn goes down, I can still access the local side of the network to fix it remotely. Of course, pfsense handles firewalling on this connection as well.

I genuinely hope this is helpful.

theOtherLuke commented 2 months ago

Some new testing on my end has revealed a couple things. As long as I have the vpn installed on the edge of the network, meaning the next hop is the modem, it works fine. Of course this is taking into account the need to declare the dns server address in the container settings on the hypervisor for Fedora or Ubuntu. If I install it 1 or 2 hops further back, like behind my pfsense, it has DNS issues. I'm sure this can be resolved with some configuration on pfsense or by setting up a proxy. Personally, I deliberately place the vpn between my pfsense and the outside world so this is not an issue for me.

I have zero experience with proxies. That is definately on my list of things to learn more about. If anyone has some advice or idea, I (we) welcome the help.

Even better would be someone who could confirm if this is all or part of the issue.

hoemada commented 2 months ago

I've the same kind of configuration :

ONT => OPNSENSE(vm in proxmox server1) => vmbr0 (vlan6)=>Routeur Wifi=> Port eth3 vlan access (vlan6) => Server Proxmox 2 vmbr0 => (WAN) Lxc RouterNordVPN (LAN)=> vmbr1 vlan7 => LXC servers in vlan 7(DHCP)

My dns is on vlan 6, to be working with the router nordvpn i had on opnsense to create a gateway + route.

nielsnetsec commented 2 months ago

Hi,

Sorry for the late response!

Ofcourse I deleted iptables and configured a clean one while VPN active, then rebooted it but somehow it did not change the contents. I will try again this week and update you on the effects, probably I am not flushing or reloading correctly i presume at this moment.

Concerning DNS, I'll try that when I can ping IP. Thusfar im not worried about DNS. When all is working, configuring this correctly is next point of interest.

I will let you know my progress!

theOtherLuke commented 2 months ago

@nielsnetsec be First off I want to apologize for not catching this earlier. I didn't notice you have duplicate rules in your iptables. This is because NordVPN was connected when the rules were saved. It's important to have NordVPN disconnected and kill switch off when you save your forwarding rules if you are entering them at the command line. The safer way is to copy the rules from the config files and paste them into your rules file. Of course make sure you change the interface to your lan interface.

theOtherLuke commented 1 month ago

@nielsnetsec I have had inconsistent success with setting up new nord test routers. I have noticed a couple things while creating a vm version using ubuntu so i can easily use wifi for my wan and tether my home network to my phone until I can get internet connected at my new house since I have unlimited tethering.

  1. It seems lan-discovery needs to be enabled nordvpn set lan-discovery on
  2. It also seems we need a mangle rule to mark our LAN traffic so nord's rules don't drop our forwarded traffic. Add the following to the top of /etc/iptables/rules.v4 changing the interface name to match your LAN interface. 
    *mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -i enp6s18 -m comment --comment our_marker -j CONNMARK --set-xmark 0xe1f1/0xffffffff
COMMIT

I had the mangle rule in the writeup previously, but removed it because I didn't see the nord DROP rules on the FORWARD chain before.

I am now running the ubuntu setup with wifi wan through my phone. I will continue to test.

theOtherLuke commented 1 month ago

@nielsnetsec @hoemada I have made progress on the dns issue. It seems dns must be setup a certain way on any router between the nordvpn instance and your network. I was always able to connect a test system directly to the nord instance but had trouble when inserting a firewall in the path. I didn't have this issue with my previous setup. I was able to figure out the fix in pfsense. I'm sure there is a similar solution in opnsense, openwrt, etc. Let me know if this helps.

...a week since the last comment Ok. Sooooooo.... I have been banging my head on the wall over you guessed it...a DNS issue.

I got my hotspot-to-home-network nordvpn vm working. I went with a vm instead of a container just because it's easier to pass the wifi that way for me. There may be an easy way to do it on a container, but this is temporary and I don't see myself going this route again. Still waiting for the provider to get cable to my house though.

Step 2 - insert a pfsense instance between the nord vm and my network. I got pfsense working just fine. I was even able to ping google.com and debian.org etc. from the lan interface inside pfsense web ui. However, I had no such luck on ANY test machine, vm, or lxc I put behind it.

The Solution -

Perusing the pfsense documentation led me to enable DNS Forwarding. This can be done under the Services > DNS Resolver or using the DNS Forwarder, but only one of those can be active at any time. If you want to declare a dns server address, like for your adguard or pihole, do it under System > General Setup. After I did this, I have not experienced any more dns issues behind my pfsense. Using Diagnostics > DNS Lookup in pfsense web ui shows it is hitting all my dns servers correctly.

I thought it broke again while writing this, but it was my phone dropping the connection LOL

On another note, I apologize for the chaos I caused restructuring the repository. I should have waited until after I was settled in and had time and reliable internet service to tackle such an undertaking. The "universal" script is not ready for production. Use at your own risk. I have reposted a slightly cleaned up version of the original write-up in the debian folder and a version modified for ubuntu in the ubuntu folder.

theOtherLuke commented 3 weeks ago

@nielsnetsec Just checking in to see if you have made any progress.