No connectivity through after VPN connect.

Trakkasure commented 2 years ago

Describe the bug

Using a portainer stack, I cannot connect (ping or wget) to any site, even within a shell inside the container.

To Reproduce using docker-compose

version: "3"
services:
  nordvpn:
    container_name: nordvpn
    hostname: nordvpn
    image: ghcr.io/bubuntux/nordlynx:latest
    cap_add:
      - NET_ADMIN
    sysctls:
      - "net.ipv4.conf.all.src_valid_mark=1"
      - "net.ipv6.conf.all.disable_ipv6=1"
    environment:
      - "PRIVATE_KEY=${PRIVATE_KEY}"
      - "ALLOWED_IPS=0.0.0.0/0"
      - "TZ=America/Chicago"
    mem_limit: "256m"
    memswap_limit: "256m"
    restart: unless-stopped

Expected behavior

After starting, I should be able to open a "console" (/bin/bash) within the container to execute "ping 1.1.1.1" all packets are dropped.

Logs

No errors in logs, but here are the logs:

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 00-firewall: executing... 
[2022-07-07T03:05:36+00:00] Firewall is up, everything has to go through the vpn
[cont-init.d] 00-firewall: exited 0.
[cont-init.d] 01-envfile: executing... 
[cont-init.d] 01-envfile: exited 0.
[cont-init.d] 01-migrations: executing... 
[migrations] started
[migrations] no migrations found
[cont-init.d] 01-migrations: exited 0.
[cont-init.d] 02-tamper-check: executing... 
[cont-init.d] 02-tamper-check: exited 0.
[cont-init.d] 10-adduser: executing... 
usermod: no changes

-------------------------------------
          _         ()
         | |  ___   _    __
         | | / __| | |  /  \
         | | \__ \ | | | () |
         |_| |___/ |_|  \__/

Brought to you by linuxserver.io
-------------------------------------

To support the app dev(s) visit:
Bubuntux: https://github.com/sponsors/bubuntux
WireGuard: https://www.wireguard.com/donations/
To support LSIO projects visit:
https://www.linuxserver.io/donate/
-------------------------------------
GID/UID
-------------------------------------

User uid:    911
User gid:    911
-------------------------------------

[cont-init.d] 10-adduser: exited 0.
[cont-init.d] 10-validate: executing... 
[cont-init.d] 10-validate: exited 0.
[cont-init.d] 20-inet: executing... 
[2022-07-07T03:05:36+00:00] Enabling connection to eth0 10.18.1.2/30
[2022-07-07T03:05:36+00:00] Enabling connection to eth1 10.19.0.2/24
[2022-07-07T03:05:36+00:00] Enabling connection to secure interfaces
[cont-init.d] 20-inet: exited 0.
[cont-init.d] 20-inet6: executing... 
[2022-07-07T03:05:37+00:00] No interface network6 detected
[cont-init.d] 20-inet6: exited 0.
[cont-init.d] 30-route: executing... 
[cont-init.d] 30-route: exited 0.
[cont-init.d] 30-route6: executing... 
[cont-init.d] 30-route6: exited 0.
[cont-init.d] 40-allowlist: executing... 
[cont-init.d] 40-allowlist: exited 0.
[cont-init.d] 90-custom-folders: executing... 
[cont-init.d] 90-custom-folders: exited 0.
[cont-init.d] 99-custom-files: executing... 
[custom-init] no custom files found exiting...
[cont-init.d] 99-custom-files: exited 0.
[cont-init.d] done.
[services.d] starting services
[2022-07-06T22:05:37-05:00] Finding the best server...
[services.d] done.
[2022-07-06T22:05:37-05:00] Using server: {
  "id": 989441,
  "created_at": "2022-05-03 10:41:58",
  "updated_at": "2022-07-07 03:02:22",
  "name": "United States #9702",
  "station": "181.214.196.68",
  "ipv6_station": "",
  "hostname": "us9702.nordvpn.com",
  "load": 4,
  "status": "online",
  "cpt": 0,
  "locations": [
    {
      "id": 79,
      "created_at": "2017-06-15 14:06:47",
      "updated_at": "2017-06-15 14:06:47",
      "latitude": 32.7833333,
      "longitude": -96.8,
      "country": {
        "id": 228,
        "name": "United States",
        "code": "US",
        "city": {
          "id": 9080300,
          "name": "Dallas",
          "latitude": 32.7833333,
          "longitude": -96.8,
          "dns_name": "dallas",
          "hub_score": 0
        }
      }
    }
  ],
  "groups": [
    {
      "id": 11,
      "created_at": "2017-06-13 13:43:00",
      "updated_at": "2017-06-13 13:43:00",
      "title": "Standard VPN servers",
      "identifier": "legacy_standard",
      "type": {
        "id": 3,
        "created_at": "2017-06-13 13:40:17",
        "updated_at": "2017-06-13 13:40:23",
        "title": "Legacy category",
        "identifier": "legacy_group_category"
      }
    },
    {
      "id": 15,
      "created_at": "2017-06-13 13:43:38",
      "updated_at": "2017-06-13 13:43:38",
      "title": "P2P",
      "identifier": "legacy_p2p",
      "type": {
        "id": 3,
        "created_at": "2017-06-13 13:40:17",
        "updated_at": "2017-06-13 13:40:23",
        "title": "Legacy category",
        "identifier": "legacy_group_category"
      }
    },
    {
      "id": 21,
      "created_at": "2017-10-27 14:23:03",
      "updated_at": "2017-10-30 08:09:48",
      "title": "The Americas",
      "identifier": "the_americas",
      "type": {
        "id": 5,
        "created_at": "2017-10-27 14:16:30",
        "updated_at": "2017-10-27 14:16:30",
        "title": "Regions",
        "identifier": "regions"
      }
    }
  ],
  "specifications": [
    {
      "id": 8,
      "title": "Version",
      "identifier": "version",
      "values": [
        {
          "id": 257,
          "value": "2.1.0"
        }
      ]
    }
  ],
  "ips": [
    {
      "id": 545723,
      "created_at": "2022-05-03 10:57:57",
      "updated_at": "2022-05-03 10:57:57",
      "server_id": 989441,
      "ip_id": 8847077,
      "type": "entry",
      "ip": {
        "id": 8847077,
        "ip": "181.214.196.68",
        "version": 4
      }
    }
  ]
}
[2022-07-06T22:05:37-05:00] Connecting...
[#] 
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add 10.5.0.2/32 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] wg set wg0 fwmark 51820
[#] ip -4 route add 0.0.0.0/0 dev wg0 table 51820
[#] ip -4 rule add not fwmark 51820 table 51820
[#] ip -4 rule add table main suppress_prefixlength 0
[#] iptables-restore -n
[#] 
[2022-07-06T22:05:37-05:00] Connected! \(ᵔᵕᵔ)/

Additional context

Proxmox VM - Alpine Linux Kernel 5.15.16 Portainer 2.14.0 Docker 20.10.12

I can add "NET_LOCAL" setting and I'm able to ping IPs within that network.

ip route:

default via 10.18.1.1 dev eth0 
10.10.10.8 via 10.18.1.1 dev eth0 
10.18.1.0/30 dev eth0 proto kernel scope link src 10.18.1.2 
172.19.0.0/24 dev eth1 proto kernel scope link src 10.19.0.2

wg show

interface: wg0
  public key: HIDDEN
  private key: (hidden)
  listening port: 51820
  fwmark: 0xca6c

peer: HIDDEN
  endpoint: 181.214.196.68:51820
  allowed ips: 0.0.0.0/0
  transfer: 0 B received, 33.39 KiB sent
  persistent keepalive: every 25 seconds

iptables -L When running this, it took 5 seconds per line to show

Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  eth0   any     10.18.1.0/30        anywhere            
    0     0 ACCEPT     all  --  eth1   any     10.19.0.0/24        anywhere            
   29  9845 ACCEPT     all  --  any    any     anywhere             anywhere             ctstate RELATED,ESTABLISHED
   33  2251 ACCEPT     all  --  lo     any     anywhere             anywhere            

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  eth0   any     anywhere             10.18.1.0/30       
    0     0 ACCEPT     all  --  eth0   any     10.18.1.0/30        anywhere            
    0     0 ACCEPT     all  --  eth1   any     anywhere             10.19.0.0/24       
    0     0 ACCEPT     all  --  eth1   any     10.19.0.0/24        anywhere            
    0     0 ACCEPT     all  --  any    any     anywhere             anywhere             ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  lo     any     anywhere             anywhere            
    0     0 ACCEPT     all  --  eth0   any     anywhere             10.10.10.8          

Chain OUTPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  any    eth0    anywhere             10.18.1.0/30       
    0     0 ACCEPT     all  --  any    eth1    anywhere             10.19.0.0/24       
   24  2066 ACCEPT     all  --  any    any     anywhere             anywhere             ctstate RELATED,ESTABLISHED
   33  2251 ACCEPT     all  --  any    lo      anywhere             anywhere            
   56  3880 ACCEPT     all  --  any    wg+     anywhere             anywhere            
    5   311 ACCEPT     udp  --  any    eth0    anywhere             anywhere             udp dpt:domain
  137 24112 ACCEPT     udp  --  any    eth0    anywhere             anywhere             udp dpt:51820
    0     0 ACCEPT     tcp  --  any    eth0    anywhere             104.17.50.74         tcp dpt:https
    2    60 ACCEPT     tcp  --  any    eth0    anywhere             104.17.49.74         tcp dpt:https

Trakkasure commented 2 years ago

Update:

After container restart, it started working. No changes to the configuration or network. However, the endpoint did automatically change after restarting the container. I did try setting my own endpoint before posting, but it did not make a difference.

Can an additional health-check be added to include pinging a remote target (set by environment variable) to confirm that packets are actually getting out and back?

Trakkasure commented 2 years ago

FYI: it did stop working again. I resolve this longer term my modifying the script to allow me to choose which server returned by setting an index.

FaizVisram commented 2 years ago

I'm having the same issue, but restarting doesn't help. This is my most minimalist compose using WSL2 on Windows:

version: "3"
services:
  nordlynx:
    image: ghcr.io/bubuntux/nordlynx
    container_name: nordlynx
    cap_add:
      - NET_ADMIN
    environment:
      - PRIVATE_KEY=$NORD_PRIVATE_KEY
    restart: unless-stopped

Logs:

[2022-07-29T18:53:13+00:00] Connecting...
[#] 
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add 10.5.0.2/32 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] wg set wg0 fwmark 51820
[#] ip -4 route add 0.0.0.0/0 dev wg0 table 51820
[#] ip -4 rule add not fwmark 51820 table 51820
[#] ip -4 rule add table main suppress_prefixlength 0
[#] iptables-restore -n
iptables-restore v1.8.7 (legacy): unknown option "--save-mark"
Error occurred at line: 5
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
[#] ip -4 rule delete table 51820
[#] ip -4 rule delete table main suppress_prefixlength 0
[#] ip link delete dev wg0
[2022-07-29T18:53:14+00:00] Connected! \(ᵔᵕᵔ)/

Ownlt commented 2 years ago

I remove the ALLOWED_IPS and i was eable to curl ifconfig.io and everything work

Trakkasure commented 2 years ago

If I set the END_POINT variable, everything works. There could be some similarities with the root cause. Lacking focus is keeping me from being able to properly diagnose this issue. If anyone has been successful in getting this to function as a gateway (not using container networking mode) I would be interested to learn more.

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

digitalhank commented 2 years ago

Experiencing the same issue. Logs says it's connected but cannot run a ping from inside the container. Same thing with the nordvpn image.

ren0d1 commented 2 years ago

Same here, even though it says "Connected! (ᵔᵕᵔ)/", I cannot contact anything outside the local network. I tried everything I could find on this repository, but none of the proposed solutions solved it.

With and without: sysctls:

net.ipv4.conf.all.rp_filter=2
net.ipv4.conf.all.src_valid_mark=1 cap_add:
SYS_ADMIN devices:
/dev/net/tun

as well as removing the ALLOWED_IPS

digitalhank commented 2 years ago

I would suggest moving to gluetun. Easy to setup and i haven't had any issues since i started using it.

lfilho commented 2 years ago

Same problem here. Using the image's last version and restarted the container a couple of times as well. @bubuntux Julio, any ideas what might be happening?

bubuntux commented 2 years ago

I think is a DNS issue created by this https://github.com/bubuntux/nordlynx/commit/94a29a6bf7a3b84cfda382e915b4a4d845de93f4 seems like you need to set a custom dns

bubuntux commented 2 years ago

If gluten or similar work for you, please use that instead if you are not willing to put the work of solving your issue with this container

lfilho commented 2 years ago

@bubuntux not sure what made you say i'm "not willing to put the work"... I tried the suggested steps so far on the thread, as well updating my containers. I don't have the knowledge to help at the code level am willing to test and help debug whichever way i can.

What do you mean setting a custom DNS? Doesnt the commit you link does exactly that (uses NordVPN's DNS by default if none is provided)? Do you mean we should be setting a custom DNS different that Nord's?

bubuntux commented 2 years ago

@lfilho i didn't meant that for you in particular, i mentioned due other ppl suggestion an alternative and i think they should use that alternative, in general if there is another solution that works just use that, if you want to make this container work for you well is up to you, i created this project for myself and it works for me, is available for anyone to use or fork to their needs, but i don't have the time to fix other ppls issues, sry.

yeah i think this issue has happen before and was the reason i removed Nordvpn DNS in the past, but that created dns leaks in some scenarios, you can try rolling back that change and build and test it

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

bubuntux / nordlynx