search

qdm12 / gluetun

VPN client in a thin Docker container for multiple VPN providers, written in Go, and using OpenVPN or Wireguard, DNS over TLS, with a few proxy servers built-in.
https://hub.docker.com/r/qmcgaw/gluetun
MIT License
8.17k stars 375 forks source link

Feature request: run nested gluetun with Wireguard #602

Open John2941 opened 3 years ago

John2941 commented 3 years ago

Is this urgent?: No

Host OS (approximate answer is fine too): Ubuntu 18

CPU arch or device name: amd64

What VPN provider are you using: Mullavad

What are you using to run your container?: Docker Compose

What is the version of the program docker-compose 1.24 // latest pull of gluetun

Running version latest built on 2020-03-13T01:30:06Z (commit d0f678c)

I am unable to host more than 1 running gluetun container (configured for wireguard) at a time because of when the container tries to add a ip route via the firewallMark, it already exists. I looked through the routing code to see if you could manually set the firewallMark but wasn't able to find an answer.

That feature doesn't work

Share your logs... (careful to remove in example tokens)


2021/09/05 12:25:20 ERROR vpn: cannot add route for interface: file exists: when adding route: {Ifindex: 15 Dst: 0.0.0.0/0 Src: <nil> Gw: <nil> Flags: [] Table: 51820}
qdm12 commented 3 years ago

Ok I'll find a way to fix that. For now maybe try setting WIREGUARD_INTERFACE=wg1 to use wg1 instead of wg0? If that works, maybe a better solution would be to use a different interface name if it already exists 🤔

John2941 commented 3 years ago

I have tried changing up every environment variable I could and it didn't change the firewall mark. Thanks for looking into it.

On Sun, Sep 5, 2021, 1:35 PM Quentin McGaw @.***> wrote:

Ok I'll find a way to fix that. For now maybe try setting WIREGUARD_INTERFACE=wg1 to use wg1 instead of wg0? If that works, maybe a better solution would be to use a different interface name if it already exists 🤔

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/qdm12/gluetun/issues/602#issuecomment-913195702, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACD56ZWBXPOWSA7DN4Y4OJLUAOS7JANCNFSM5DO7ACMA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

qdm12 commented 3 years ago

Can you share your configuration (remove private key/credentials)? If you can share logs too that would help.

I just tried running two gluetun instances with Wireguard both on a Linux host and on Docker desktop on windows and it doesn't error at all.

John2941 commented 3 years ago

Sorry I didn't provide more information as to replicating the bug since I was trying to do something a little different. I was able to create 2 different, independent version of gluetun by themselves, but when I use try to use the first gluetun container as the network stack for the second one I get the error. Heres my docker-compose.yml where I was able to replicate. In order to start gluetun2 you have to have gluetun started since I haven't worked out the dependency error via docker-compose yet. To get gluetun started via this docker-compose.yml you'll have to change the network_mode for gluetun2 to

networks:

then change the yml back to the network_mode: "container:gluetun" and itll start it up and give the fwmark error.

version: "3.6"
services:
  gluetun:
    image: qmcgaw/gluetun
    container_name: gluetun
    cap_add:
      - NET_ADMIN
    #ports:
    #  - 8888:8888/tcp # HTTP proxy
    #  - 8388:8388/tcp # Shadowsocks
    #  - 8388:8388/udp # Shadowsocks
    #  - 8000:8000/tcp # Built-in HTTP control server
    # command:
    networks:
      - gluetun
    volumes:
      - /opt/gluetun/config:/gluetun
    environment:
      # More variables are available, see the Wiki table
      - COUNTRY=USA
      - VPNSP=mullvad
      - VPN_TYPE=wireguard
      - WIREGUARD_INTERFACE=wg0
      - TZ=America/New_York
      - WIREGUARD_PORT=51820
      - WIREGUARD_ADDRESS=X.X.X.X/XX
      - WIREGUARD_PRIVATE_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      - DOT=on
      - DOT_PROVIDERS=cloudflare
    restart: always

  gluetun2:
    image: qmcgaw/gluetun
    container_name: gluetun2
    cap_add:
      - NET_ADMIN
    #ports:
    #  - 8888:8888/tcp # HTTP proxy
    #  - 8388:8388/tcp # Shadowsocks
    #  - 8388:8388/udp # Shadowsocks
    #  - 8000:8000/tcp # Built-in HTTP control server
    # command:
    network_mode: "container:gluetun"
    volumes:
      - /opt/gluetun/config2:/gluetun
    environment:
      # More variables are available, see the Wiki table
      - COUNTRY=USA
      - VPNSP=mullvad
      - VPN_TYPE=wireguard
      - WIREGUARD_INTERFACE=wg1
      - TZ=America/New_York
      - WIREGUARD_PORT=51820
      - WIREGUARD_ADDRESS=X.X.X.X/XX
      - WIREGUARD_PRIVATE_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      - DOT=on
      - DOT_PROVIDERS=cloudflare
    restart: always

networks:
  gluetun:

Logs for gluetun2:

========================================,
,
Running version latest built on 2021-09-05T22:58:18Z (commit f9cb710),
,
📣 Wireguard is now supported for Mullvad, IVPN and Windscribe!,
,
🔧 Need help? https://github.com/qdm12/gluetun/discussions/new,
🐛 Bug? https://github.com/qdm12/gluetun/issues/new,
✨ New feature? https://github.com/qdm12/gluetun/issues/new,
☕ Discussion? https://github.com/qdm12/gluetun/discussions/new,
💻 Email? quentin.mcgaw@gmail.com,
💰 Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12,
2021/09/06 10:26:09 INFO Alpine version: 3.14.2,
2021/09/06 10:26:09 INFO OpenVPN 2.4 version: 2.4.11,
2021/09/06 10:26:09 INFO OpenVPN 2.5 version: 2.5.2,
2021/09/06 10:26:09 INFO Unbound version: 1.13.2,
2021/09/06 10:26:09 INFO IPtables version: v1.8.7,
2021/09/06 10:26:09 INFO Settings summary below:,
|--VPN:,
   |--Type: wireguard,
   |--Wireguard:,
      |--Network interface: wg1,
      |--Private key is set,
      |--Addresses: ,
         |--X.X.X.X/XX,
   |--Mullvad settings:,
      |--Countries: usa,
      |--Wireguard selection:,
         |--Custom port: 51820,
|--DNS:,
   |--Plaintext address: 1.1.1.1,
   |--DNS over TLS:,
      |--Unbound:,
          |--DNS over TLS providers:,
              |--Cloudflare,
          |--Listening port: 53,
          |--Access control:,
              |--Allowed:,
                  |--0.0.0.0/0,
                  |--::/0,
          |--Caching: enabled,
          |--IPv4 resolution: enabled,
          |--IPv6 resolution: disabled,
          |--Verbosity level: 1/5,
          |--Verbosity details level: 0/4,
          |--Validation log level: 0/2,
          |--Username: ,
      |--Blacklist:,
         |--Blocked categories: malicious,
         |--Additional IP networks blocked: 13,
      |--Update: every 24h0m0s
|--Firewall:
|--Log:
   |--Level: INFO
|--System:
   |--Process user ID: 1000
   |--Process group ID: 1000
   |--Timezone: america/new_york
|--Health:
   |--Server address: 127.0.0.1:9999
   |--OpenVPN:
      |--Initial duration: 6s
      |--Addition duration: 5s
|--HTTP control server:
   |--Listening port: 8000
   |--Logging: enabled
|--Public IP getter:
   |--Fetch period: 12h0m0s
   |--IP file: /tmp/gluetun/ip
|--Github version information: enabled
2021/09/06 10:26:09 INFO routing: default route found: interface eth0, gateway 10.10.10.1
2021/09/06 10:26:09 INFO routing: local ethernet link found: eth0
2021/09/06 10:26:09 INFO routing: local ipnet found: 10.10.10.0/24
2021/09/06 10:26:09 INFO routing: default route found: interface eth0, gateway 10.10.10.1
2021/09/06 10:26:09 INFO routing: adding route for 0.0.0.0/0
2021/09/06 10:26:09 INFO firewall: firewall disabled, only updating allowed subnets internal list
2021/09/06 10:26:09 INFO routing: default route found: interface eth0, gateway 10.10.10.1
2021/09/06 10:26:09 INFO TUN device is not available: open /dev/net/tun: no such file or directory; creating it...
2021/09/06 10:26:09 INFO firewall: enabling...
2021/09/06 10:26:09 INFO firewall: enabled successfully
2021/09/06 10:26:09 INFO healthcheck: listening on 127.0.0.1:9999
2021/09/06 10:26:09 INFO dns over tls: using plaintext DNS at address 1.1.1.1
2021/09/06 10:26:09 INFO http server: listening on :8000
2021/09/06 10:26:09 ERROR healthcheck: listen tcp 127.0.0.1:9999: bind: address already in use
2021/09/06 10:26:09 INFO firewall: setting VPN connection through firewall...
2021/09/06 10:26:09 ERROR vpn: cannot add route for interface: file exists: when adding route: {Ifindex: 3 Dst: 0.0.0.0/0 Src: <nil> Gw: <nil> Flags: [] Table: 51820}
2021/09/06 10:26:09 INFO vpn: retrying in 15s
2021/09/06 10:26:09 INFO healthcheck: healthy!
2021/09/06 10:26:24 INFO firewall: setting VPN connection through firewall...
2021/09/06 10:26:24 ERROR vpn: cannot add route for interface: file exists: when adding route: {Ifindex: 4 Dst: 0.0.0.0/0 Src: <nil> Gw: <nil> Flags: [] Table: 51820}
2021/09/06 10:26:24 INFO vpn: retrying in 30s
2021/09/06 10:26:54 INFO firewall: setting VPN connection through firewall...
2021/09/06 10:26:54 ERROR vpn: cannot add route for interface: file exists: when adding route: {Ifindex: 5 Dst: 0.0.0.0/0 Src: <nil> Gw: <nil> Flags: [] Table: 51820}
2021/09/06 10:26:54 INFO vpn: retrying in 1m0s
2021/09/06 10:27:54 INFO firewall: setting VPN connection through firewall...
2021/09/06 10:27:54 ERROR vpn: cannot add route for interface: file exists: when adding route: {Ifindex: 6 Dst: 0.0.0.0/0 Src: <nil> Gw: <nil> Flags: [] Table: 51820}
2021/09/06 10:27:54 INFO vpn: retrying in 2m0s
2021/09/06 10:29:54 INFO firewall: setting VPN connection through firewall...
2021/09/06 10:29:54 ERROR vpn: cannot add route for interface: file exists: when adding route: {Ifindex: 7 Dst: 0.0.0.0/0 Src: <nil> Gw: <nil> Flags: [] Table: 51820}
2021/09/06 10:29:54 INFO vpn: retrying in 4m0s
2021/09/06 10:33:54 INFO firewall: setting VPN connection through firewall...
2021/09/06 10:33:54 ERROR vpn: cannot add route for interface: file exists: when adding route: {Ifindex: 8 Dst: 0.0.0.0/0 Src: <nil> Gw: <nil> Flags: [] Table: 51820}
2021/09/06 10:33:54 INFO vpn: retrying in 8m0s
qdm12 commented 3 years ago

Oh boy! Sorry to disappoint, but Gluetun is really really not designed for such use case.

There would be quite a bit of gymnastics to do firewall and routing wise (on top of that firewall mark conflicting) which I don't really have the time to address right now (and it would most likely break other things if I do now).

But let's leave the issue opened, I might get to it in the future!

ShayBox commented 1 year ago

Progress update in 2023:

docker-compose.yml

version: "3"
services:
  gluetun-windscribe:
    image: qmcgaw/gluetun
    container_name: gluetun-windscribe
    # line above must be uncommented to allow external containers to connect.
    # See https://github.com/qdm12/gluetun/wiki/Connect-a-container-to-gluetun#external-container-to-gluetun
    cap_add:
      - NET_ADMIN
    devices:
      - /dev/net/tun:/dev/net/tun
    ports:
      - 8888:8888/tcp # HTTP proxy (Windscribe)
      - 8889:8889/tcp # HTTP proxy (Cloudflare)
      - 8388:8388/tcp # Shadowsocks (Windscribe)
      - 8389:8389/tcp # Shadowsocks (Cloudflare)
      - 8388:8388/udp # Shadowsocks (Windscribe)
      - 8389:8389/udp # Shadowsocks (Cloudflare)
    volumes:
      - C:\Users\..\Gluetun\Windscribe:/gluetun
    environment:
      # See https://github.com/qdm12/gluetun/wiki
      - VPN_SERVICE_PROVIDER=windscribe
      - VPN_TYPE=wireguard
      # Wireguard:
      - WIREGUARD_PRIVATE_KEY=X
      - WIREGUARD_ADDRESSES=X.X.X.X/X
      - WIREGUARD_PRESHARED_KEY=X
      - SERVER_REGIONS=X
      # Timezone for accurate log times
      - TZ=America/New_York
      # Server list updater. See https://github.com/qdm12/gluetun/wiki/Updating-Servers#periodic-update
      - UPDATER_PERIOD=24h
      # Proxy
      - HTTPPROXY=on
      - HTTPPROXY_LOG=on
      - SHADOWSOCKS=on
      - SHADOWSOCKS_LOG=on
  gluetun-cloudflare:
    image: qmcgaw/gluetun
    container_name: gluetun-cloudflare
    # line above must be uncommented to allow external containers to connect.
    # See https://github.com/qdm12/gluetun/wiki/Connect-a-container-to-gluetun#external-container-to-gluetun
    cap_add:
      - NET_ADMIN
    devices:
      - /dev/net/tun:/dev/net/tun
    network_mode: "service:gluetun-windscribe"
    volumes:
      - C:\Users\..\Gluetun\Cloudflare:/gluetun
    environment:
      # See https://github.com/qdm12/gluetun/wiki
      - VPN_SERVICE_PROVIDER=custom
      - VPN_TYPE=wireguard
      # Wireguard:
      - VPN_ENDPOINT_IP=X.X.X.X
      - VPN_ENDPOINT_PORT=X
      - WIREGUARD_PUBLIC_KEY=X
      - WIREGUARD_PRIVATE_KEY=X
      - WIREGUARD_ADDRESSES=X.X.X.X/X
      # Timezone for accurate log times
      - TZ=America/New_York
      # Server list updater. See https://github.com/qdm12/gluetun/wiki/Updating-Servers#periodic-update
      - UPDATER_PERIOD=24h
      # Proxy
      - HTTPPROXY=on
      - HTTPPROXY_LOG=on
      - HTTPPROXY_LISTENING_ADDRESS=:8889
      - SHADOWSOCKS=on
      - SHADOWSOCKS_LOG=on
      - SHADOWSOCKS_LISTENING_ADDRESS=:8389
      # HTTP Server
      - HTTP_CONTROL_SERVER_ADDRESS=:8001

The Windscribe container starts successfully, but the Cloudflare container still has issues

Running version latest built on 2023-05-10T09:05:10.273Z (commit 4bb77eb)

INFO [routing] default route found: interface eth0, gateway 172.23.0.1, assigned IP 172.23.0.2 and family v4
INFO [routing] local ethernet link found: eth0
INFO [routing] local ipnet found: 172.23.0.0/16
INFO [firewall] enabling...
INFO [firewall] enabled successfully
INFO [storage] merging by most recent 13056 hardcoded servers and 13056 servers read from /gluetun/servers.json
INFO Alpine version: 3.17.3
INFO OpenVPN 2.4 version: 2.4.12
INFO OpenVPN 2.5 version: 2.5.8
INFO Unbound version: 1.17.1
INFO IPtables version: v1.8.8
INFO Settings summary:
├── VPN settings:
|   ├── VPN provider settings:
|   |   ├── Name: custom
|   |   └── Server selection settings:
|   |       ├── VPN type: wireguard
|   |       ├── Target IP address: X.X.X.X
|   |       └── Wireguard selection settings:
|   |           ├── Endpoint IP address: X.X.X.X
|   |           ├── Endpoint port: X
|   |           └── Server public key: X
|   └── Wireguard settings:
|       ├── Private key: X
|       ├── Interface addresses:
|       |   └── X.X.X.X/X
|       └── Network interface: tun0
├── DNS settings:
|   ├── DNS server address to use: 127.0.0.1
|   ├── Keep existing nameserver(s): no
|   └── DNS over TLS settings:
|       ├── Enabled: yes
|       ├── Update period: every 24h0m0s
|       ├── Unbound settings:
|       |   ├── Authoritative servers:
|       |   |   └── cloudflare
|       |   ├── Caching: yes
|       |   ├── IPv6: no
|       |   ├── Verbosity level: 1
|       |   ├── Verbosity details level: 0
|       |   ├── Validation log level: 0
|       |   ├── System user: root
|       |   └── Allowed networks:
|       |       ├── 0.0.0.0/0
|       |       └── ::/0
|       └── DNS filtering settings:
|           ├── Block malicious: yes
|           ├── Block ads: no
|           ├── Block surveillance: no
|           └── Blocked IP networks:
|               ├── 127.0.0.1/8
|               ├── 10.0.0.0/8
|               ├── 172.16.0.0/12
|               ├── 192.168.0.0/16
|               ├── 169.254.0.0/16
|               ├── ::1/128
|               ├── fc00::/7
|               ├── fe80::/10
|               ├── ::ffff:127.0.0.1/104
|               ├── ::ffff:10.0.0.0/104
|               ├── ::ffff:169.254.0.0/112
|               ├── ::ffff:172.16.0.0/108
|               └── ::ffff:192.168.0.0/112
├── Firewall settings:
|   └── Enabled: yes
├── Log settings:
|   └── Log level: INFO
├── Health settings:
|   ├── Server listening address: 127.0.0.1:9999
|   ├── Target address: cloudflare.com:443
|   ├── Duration to wait after success: 5s
|   ├── Read header timeout: 100ms
|   ├── Read timeout: 500ms
|   └── VPN wait durations:
|       ├── Initial duration: 6s
|       └── Additional duration: 5s
├── Shadowsocks server settings:
|   ├── Enabled: yes
|   ├── Listening address: :8389
|   ├── Cipher: chacha20-ietf-poly1305
|   ├── Password: [not set]
|   └── Log addresses: yes
├── HTTP proxy settings:
|   ├── Enabled: yes
|   ├── Listening address: :8889
|   ├── User:
|   ├── Password: [not set]
|   ├── Stealth mode: no
|   ├── Log: yes
|   ├── Read header timeout: 1s
|   └── Read timeout: 3s
├── Control server settings:
|   ├── Listening address: :8001
|   └── Logging: yes
├── OS Alpine settings:
|   ├── Process UID: 1000
|   ├── Process GID: 1000
|   └── Timezone: America/New_York
├── Public IP settings:
|   ├── Fetching: every 12h0m0s
|   └── IP file path: /tmp/gluetun/ip
└── Version settings:
    └── Enabled: yes
INFO [routing] default route found: interface eth0, gateway 172.23.0.1, assigned IP 172.23.0.2 and family v4
INFO [routing] adding route for 0.0.0.0/0
INFO [firewall] setting allowed subnets...
INFO [routing] default route found: interface eth0, gateway 172.23.0.1, assigned IP 172.23.0.2 and family v4
INFO [dns over tls] using plaintext DNS at address 1.1.1.1
INFO [http proxy] listening on :8889
INFO [http server] http server listening on [::]:8001
INFO [healthcheck] listening on 127.0.0.1:9999
INFO [shadowsocks] listening TCP on :8389
INFO [shadowsocks] listening UDP on :8389
INFO [wireguard] Using available kernelspace implementation
ERROR [vpn] cannot add Wireguard link: file exists
INFO [vpn] retrying in 15s
INFO [healthcheck] healthy!
INFO [firewall] allowing VPN connection...
INFO [wireguard] Using available kernelspace implementation
ERROR [vpn] cannot add Wireguard link: file exists
INFO [vpn] retrying in 30s
...

The ERROR [vpn] cannot add Wireguard link: file exists seems to be the final problem to overcome.

EDIT: It also appears Healthcheck conflicts on port 9999, an option to set that listening address is needed.

EDIT: My use case for this is to use a VPN provider to change geo-location, then Cloudflare Warp to connect to the closest location to that provider since Warp doesn't let you change the location manually (that I know of)

Note: The ShadowSocks wiki mislabels SHADOWSOCKS_ADDRESS, it's SHADOWSOCKS_LISTENING_ADDRESS

ShayBox commented 1 year ago

Containers sharing their network with network mode is unlikely to work without large changes, the easier way is probably to add the ability for Gluetun to connect itself to Wireguard, allowing for more flexibility than the above use case.

EDIT: #411 and #413 might be related? I don't really understand them.