bubuntux / nordlynx

GNU General Public License v3.0
209 stars 48 forks source link

Can't route traffic outside local network. #146

Closed TKinslayer closed 11 months ago

TKinslayer commented 1 year ago

Bug description

Short version : I'm having the hardest time making Nordlynx work on any Linux (VM) I set up.... and I'm failing miserably. Long version : I used to make Nordlynx docker (and NordVPN) work on Raspberri Pi and it was just fine. But I decided to get rid of it and invested in a proper Home Lab. I set up Proxmox and I've been playing with many VMs for some time and everything works but for my VPN. First, whatever Linux I was using were having problems with IP tables and the NordVPN docker. NordLynx or NordVPN were always telling me the iptables version wasn't compatible.

I tried on Fedora, Oracle, CentOS Stream, Debian... sometimes with older versions (but maybe not old enough...), but to no avail. I played with iptable-legacy and nftable (mainly followed advice from people having that same problem with Docker and iptables) but I could never make it work (I do have to say iptable/nftable are not something I'm really comfortable with at the moment, especially because it used to work on Raspbian out of the box). Having read that all those Linux version don't maintain a separate iptables-legacy binary like Debian, that's what I ended up installing. It still isn't working with the NordVPN docker, but... I don't get the error with the NordLynx. With NordLynx on Debian, the thing is... it seems to connect to the VPN server, as some of the logs show. But it doesn't route traffic through the VPN server. Depending on the Linux I set it up on, sometimes it would not route traffic outside my local network (a ping 8.8.8.8 wasn't working but a ping 192.168.1.10 would work) and sometimes it would work.

As for the docker compose, in order to minimize conflicts/bad configuration, I made all comparisons using bare settings (keeping in mind the fact I also did play around with many different settings for the last three days...).

Docker-compose

version: "3"
services:
  nordlynx:
    image: ghcr.io/bubuntux/nordlynx
    container_name: nordlynx
    cap_add:
      - NET_ADMIN
      - NET_RAW
    environment:
      - PRIVATE_KEY= [redacted]
    restart: unless-stopped

So, I guess there are several issues there, and I will try to lay them out clearly one after the others.

Number 1

That iptables seems to be missing. I get that message on nearly all Linux distributors but Debian. Something other users have mentioned, like here : Depricate the use of IPTables? and here : [System will not start]() (but the solutions given in that last one didn't seem to work for me... or I'm missing something)

Expected behavior

Well... Linux should automatically translate its nftables to iptables. Which it does with all the other dockers I run. Only NordLynx shows this problem. And I have than 40+ other dockers running without any problem.

Logs

iptables v1.8.8 (legacy): can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
iptables v1.8.8 (legacy): can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
iptables v1.8.8 (legacy): can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
[2023-08-12T20:08:03+00:00] Firewall is up, everything has to go through the vpn
[2023-08-12T20:08:03+00:00] iptables is not functional. Ensure your container config adds --cap-add=NET_RAW

Additional context

As I said, I followed many tutorials (because I'm not well versed with iptable to begin with) and I throw the towel after three days.

Number 2

I seem to connect to NordVPN. But traffic doesn't route to VPN server.

Expected behavior

When I do docker exec -it NordLynx curl ipinfo.io/$(curl ifconfig.me) - I my IP address. And when I use services like : http://checkmyip.torrentprivacy.com where you download a torrent, it shows my home IP address and not the one of the VPN server I'm supposed to be connected to (I do the same kind of test with NordBPN or NordLynx on my Raspeberri Pi and it works just fine).

Docker-compose

Here is the docker compose I use.

version: '3'

services:
  nordvpn:
    image: ghcr.io/bubuntux/nordvpn:latest
    cap_add:
      - NET_ADMIN
      - NET_RAW
    environment:
      - PRIVATE_KEY=[Redacted]
      - NET_LOCAL=192.168.1.0/24
      - TZ=[Redacted]
    ports:
      - 6881:6881
      - 6881:6881/udp
      - 8080:8080
    sysctls:
      - net.ipv6.conf.all.disable_ipv6=1 
    network_mode: "bridge"
    restart: unless-stopped

  qbittorrent:
    image: linuxserver/qbittorrent:latest
    container_name: qbittorrent
    environment:
      - PUID=1026
      - PGID=101
      - WEBUI_PORT=8080
      - TZ=Europe/Paris
      #- WEBUI_USERNAME=admin
      #- WEBUI_PASSWORD=adminadmin
    restart: unless-stopped
    volumes:
      - /var/docker/qbitorrent:/config
      - /mnt/nas/Download:/downloads
      - /mnt/nas/TV:/tv
      - /mnt/nas/Movies:/movies
    network_mode: "service:nordvpn"
    depends_on:
      - nordvpn

Logs

I don't get an error message on Debian (because with Debian, I can reach outside my network, which wouldn't work with Oracle 8 or 9 or Fedora, in fact...)

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    11  100    11    0     0     90      0 --:--:-- --:--:-- --:--:--    90
{
  "ip": "[redacted]",
  "hostname": "[redacted]",
  "city": "[redacted]",
  "region": "[redacted]",
  "country": "[redacted]",
  "loc": "[redacted]",
  "org": "[redacted].",
  "postal": "[redacted]",
  "timezone": "[redacted]",
  "readme": "https://ipinfo.io/missingauth"

Well, lots of redacted ;-) But the gist of it is : it's all my home. Even when the logs says I'm supposed to be connected to a server in another country and it doesn't route traffic... What NordLynx log shows is this :

root@pc-137:/home/myname# docker logs NordLynx
[migrations] started
[migrations] no migrations found
[2023-08-12T20:46:35+00:00] Firewall is up, everything has to go through the vpn
usermod: no changes
───────────────────────────────────────
  _____ __ __ _____ _____ _____ _____ 
 |     |  |  |   __|_   _|     |     |
 |   --|  |  |__   | | | |  |  | | | |
 |_____|_____|_____| |_| |_____|_|_|_|
       _____ __ __ _ __    ____  
      | __  |  |  | |  |  |    \ 
      | __ -|  |  | |  |__|  |  |
      |_____|_____|_|_____|____/ 

  Based on images from linuxserver.io
───────────────────────────────────────

To support the app dev(s) visit:
Bubuntux: https://github.com/sponsors/bubuntux
WireGuard: https://www.wireguard.com/donations/
To support LSIO projects visit:
https://www.linuxserver.io/donate/

───────────────────────────────────────
GID/UID
───────────────────────────────────────

User UID:    911
User GID:    911
───────────────────────────────────────

[2023-08-12T20:46:35+00:00] Enabling connection to eth0 172.17.0.3/16
[2023-08-12T20:46:35+00:00] Enabling connection to secure interfaces
[2023-08-12T20:46:35+00:00] No interface network6 detected
[custom-init] No custom files found, skipping...
[ls.io-init] done.
[2023-08-12T20:46:35+00:00] Finding the best server...
[2023-08-12T20:46:36+00:00] Using server: {
  "id": 956717,
  "created_at": "2020-03-10 12:25:54",
  "updated_at": "2023-08-12 20:41:19",
  "name": "[redacted] #543",
  "station": "217.138.207.179",
  "ipv6_station": "",
  "hostname": "fr543.nordvpn.com",
  "load": 11,
  "status": "online",
  "cpt": 0,
  "locations": [
    {
      "id": 87,
      "created_at": "2017-06-15 14:06:47",
      "updated_at": "2017-06-15 14:06:47",
      "latitude": [redacted],
      "longitude": [redacted],
      "country": {
        "id": 74,
        "name": "[redacted]",
        "code": "[redacted]",
        "city": {
          "id": 2886284,
          "name": "[redacted]",
          "latitude": [redacted],
          "longitude": [redacted],
          "dns_name": "[redacted]",
          "hub_score": 0
        }
      }
    }
  ],
  "groups": [
    {
      "id": 11,
      "created_at": "2017-06-13 13:43:00",
      "updated_at": "2017-06-13 13:43:00",
      "title": "Standard VPN servers",
      "identifier": "legacy_standard",
      "type": {
        "id": 3,
        "created_at": "2017-06-13 13:40:17",
        "updated_at": "2017-06-13 13:40:23",
        "title": "Legacy category",
        "identifier": "legacy_group_category"
      }
    },
    {
      "id": 15,
      "created_at": "2017-06-13 13:43:38",
      "updated_at": "2017-06-13 13:43:38",
      "title": "P2P",
      "identifier": "legacy_p2p",
      "type": {
        "id": 3,
        "created_at": "2017-06-13 13:40:17",
        "updated_at": "2017-06-13 13:40:23",
        "title": "Legacy category",
        "identifier": "legacy_group_category"
      }
    },
    {
      "id": 19,
      "created_at": "2017-10-27 14:17:17",
      "updated_at": "2017-10-27 14:17:17",
      "title": "Europe",
      "identifier": "europe",
      "type": {
        "id": 5,
        "created_at": "2017-10-27 14:16:30",
        "updated_at": "2017-10-27 14:16:30",
        "title": "Regions",
        "identifier": "regions"
      }
    }
  ],
  "specifications": [
    {
      "id": 8,
      "title": "Version",
      "identifier": "version",
      "values": [
        {
          "id": 257,
          "value": "2.1.0"
        }
      ]
    }
  ],
  "ips": [
    {
      "id": 966314,
      "created_at": "2023-06-27 13:03:35",
      "updated_at": "2023-06-27 13:03:35",
      "server_id": 956717,
      "ip_id": 107204,
      "type": "entry",
      "ip": {
        "id": 107204,
        "ip": "217.138.207.179",
        "version": 4
      }
    }
  ]
}
[2023-08-12T20:46:36+00:00] Connecting...
[#] 
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add 10.5.0.2/32 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] resolvconf -a wg0 -m 0 -x
[#] wg set wg0 fwmark 51820
[#] ip -4 route add 0.0.0.0/0 dev wg0 table 51820
[#] ip -4 rule add not fwmark 51820 table 51820
[#] ip -4 rule add table main suppress_prefixlength 0
[#] iptables-restore -n
[#] 
[2023-08-12T20:46:36+00:00] Connected! \(ᵔᵕᵔ)/

So... it says it's connected. But... it doesn't "work".

I hope I haven't confused you with my explanation. It's rather late where I am and I am a bit tired and disappointed (in myself, mainly ;-). In the end, I guess it may not come from NordLynx docker, and only from my own incompetence, but that's why I'm asking for a bit of help ;-)

Basically, do I need to spend days trying to understand iptable/nftable and how to dockerize my own version of fireguard docker for NordVPN ? (trick question, because the answer should always be YES. It's just... I don't really have the time to do that now ;-). Or.... do I need to switch to older Linux version of those distributions ? (I really don't want to keep using my raspberry pi for that) and if so, which one should I deploy ?

Johannes-m-TK commented 1 year ago

cap_add:

Add SYS_MODULE to your docker-compose.yml

cloudgprabhu commented 1 year ago

Even I have the similar suatuation ... for some reason, wireguard says my privatekey is wrong! I have copied the exact same that have popped up in the NordVPN logs. Do we need to set NordToken?

LOGS

[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
Key is not the correct length or format: `/run/secrets/privatekey'
Configuration parsing error
[#] ip link delete dev wg0
          "name": "New York",
          "latitude": 40.7141667,
          "longitude": -74.0063889,
          "dns_name": "new-york",
          "hub_score": 0
        }
      }
    }
  ],
  "groups": [
    {
      "id": 11,
      "created_at": "2017-06-13 13:43:00",
      "updated_at": "2017-06-13 13:43:00",
      "title": "Standard VPN servers",
      "identifier": "legacy_standard",
      "type": {
        "id": 3,
        "created_at": "2017-06-13 13:40:17",
        "updated_at": "2017-06-13 13:40:23",
        "title": "Legacy category",
        "identifier": "legacy_group_category"
      }
    },
    {
      "id": 15,
      "created_at": "2017-06-13 13:43:38",
      "updated_at": "2017-06-13 13:43:38",
      "title": "P2P",
      "identifier": "legacy_p2p",
      "type": {
        "id": 3,
        "created_at": "2017-06-13 13:40:17",
        "updated_at": "2017-06-13 13:40:23",
        "title": "Legacy category",
        "identifier": "legacy_group_category"
      }
    },
    {
      "id": 21,
      "created_at": "2017-10-27 14:23:03",
      "updated_at": "2017-10-30 08:09:48",
      "title": "The Americas",
      "identifier": "the_americas",
      "type": {
        "id": 5,
        "created_at": "2017-10-27 14:16:30",
        "updated_at": "2017-10-27 14:16:30",
        "title": "Regions",
        "identifier": "regions"
      }
    }
  ],
  "specifications": [
    {
      "id": 8,
      "title": "Version",
      "identifier": "version",
      "values": [
        {
          "id": 257,
          "value": "2.1.0"
        }
      ]
    }
  ],
  "ips": [
    {
      "id": 576902,
      "created_at": "2022-06-08 10:11:25",
      "updated_at": "2022-06-08 10:11:25",
      "server_id": 990689,
      "ip_id": 10198940,
      "type": "entry",
      "ip": {
        "id": 10198940,
        "ip": "191.101.160.164",
        "version": 4
      }
    }
  ]
}
[2023-10-15T09:02:36+01:00] Connecting...
[2023-10-15T09:02:37+01:00] Connected! \(ᵔᵕᵔ)/

Docker-Compose file:

version: "3"
services:
  nordlynx:
    image: ghcr.io/bubuntux/nordlynx
    container_name: nordlynx
    cap_add:
      - NET_ADMIN                             # required
      - NET_RAW                                 # maybe
      - SYS_MODULE                          # maybe
    environment:
      - PRIVATE_KEY=/run/secrets/privatekey   # required
      - QUERY=filters\[country_id\]=228
      - NET_LOCAL=192.168.1.0/16
      - ALLOWED_IPS=0.0.0.0/0
      - KILLSWITCH=Enabled
      - TABLE=auto
    secrets:
      - privatekey
    restart: unless-stopped
    ports:
      - 6980:6980
      - 51820:51820 #nordlynx
    sysctls:
      - net.ipv6.conf.all.disable_ipv6=1

secrets:
  privatekey:
    file: ./privatekey.txt
github-actions[bot] commented 11 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

RedPine404 commented 4 months ago

I had the same issue as @TKinslayer and just got this working. My container host is Rocky Linux 9.4.

Posting here in case someone else needs this:

To fix the error "can't initialize iptables table 'filter':"

Run this on the container host

sudo modprobe iptable_raw

To make it load automatically after reboot

echo "iptable_raw" | sudo tee /etc/modules-load.d/iptable_raw.conf

To fix the container not sending traffic over the VPN:

Add to your docker-compose.yml

sysctls:
      - net.ipv4.conf.all.src_valid_mark=1