Cannot access the VPN network inside a container

[devantoine]

Expected behavior

When I'm connected through a VPN I should be able to access the VPN network inside a container.

Actual behavior

When I'm connected through a VPN I cannot access the VPN network, either by hostname or by IP.

Information

Diagnostic ID: FD552713-528B-478F-9BA6-8A6528AA18A4

Docker for Mac: version: 18.03.0-ce-mac60 (dd2831d4b7421cf559a0881cc7a5fdebeb8c2b98)
macOS: version 10.13.3 (build: 17D102)
logs: /tmp/FD552713-528B-478F-9BA6-8A6528AA18A4/20180418-152905.tar.gz
[OK]     vpnkit
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     kubernetes
[OK]     files
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     disk

Steps to reproduce the behavior

1. Connect to the VPN (I'm not sure which one it is, it's the client's VPN and I'm using the Pulse Secure client)
2. Start docker-compose
3. Log into a container
4. Try to connect to one of the VPN network machine, using SSH, either by hostname or by IP
5. Wait for the connection time out.

Additional informations

This was working perfectly fine when I was using Linux with the same client.

[DiegoGallegos4]

I concur with this. For me, it works occasionally and the only way to make it work is reinstalling docker.

I tried the steps in this issue but did not work.

In my case I try to connect to a db instance in a aws vpn. In the host I try:

telnet <db_host> <port>

and succeeds connecting to it while in the docker instance I get Host unreachable or Operation Timeout.

[YRM64]

For those developers utilizing a docker-compose file, try adding additional host properties to your Docker-compose file(s).

Try:

extra_hosts:

• "internal.server.company.com: 172.162.4.1111"

This should get your Docker-machine talking to the internal network.

[cdenneen]

2628 might be related.

Work around is the following but still needs to be fixed upstream because work around is needed after reboot constantly. (https://github.com/docker/for-win/issues/1667#issuecomment-367357067)

[YRM64]

Devantoine, did you try some of the troubleshooting steps in #1803; more specifically referring to adding a UI element that allows you to customize IP address access through a range process on both edge and stable branches. Did you even see any similarities in #1803 to your use case? If so, please share the similarities/dissimilarites. Additionally, you added in your actual behavior "When I'm connecting through a VPN".........I would like to know the steps you taken to connect to the VPN. Your steps to reproduce the behavior simply states "connect to the VPN (I'm not sure which one it is...).

Diegogallegos4 indicated he'd taken the steps in #1803; I would like to see his logs, expected, and actual behavior(s), steps taken to reproduce behavior, and any other additional information he might have.

[cdenneen]

/lifecycle frozen

[mightybs]

I'm experiencing this as well on Version 18.06.1-ce-mac73 (26764) I'm not sure what version I previously had, but this was working, and now I cannot access resources on the VPN that my host is connected to. Alternatively I am able to port forward on local host to work around.

e.g. Inside container: psql -h ${vpn_ip_address} will fail, but after setting up port forwarding on my mac to ${vpn_ip_address}, e.g. ynetd -proxy ":5432 test-edgelookout-pg-20170609.cqdj8tjqocal.us-west-2.rds.amazonaws.com:5432" date that's when from inside the container: psql -h ${local_address_for_mac} does work

I've also been trying to find a working version, by installing older versions of Docker, but have not been successful in finding a working version yet.

[zhyon404]

I'm experiencing the same. and solving by change the bip setting. . docker0 using 172.17.x.x as address. so if your vpn address is in the same subnet, it will route to docker0. the above setting change the docker0 ip address. than it can rote correctly.

[mightybs]

As @zhyon404 mentioned, I ams also using the "bip" setting in the config, because I've got a network in the same default address space, despite this setting, I am unable to directly contact assets on the VPN.

But now that you've brought it up, it's got me digging further into my VPN setups to see if something looks out of whack there. I know I haven't changed VPNs or clients, but you never know. I'll do some digging and report back.

[cdenneen]

Does this only help docked or will this help the 10. addresses assigned from Kubernetes as well? On Wed, Oct 31, 2018 at 8:02 AM Brett Stauner notifications@github.com wrote:

As @zhyon404 https://github.com/zhyon404 mentioned, I ams also using the "bip" setting in the config, because I've got a network in the same default address space, despite this setting, I am unable to directly contact assets on the VPN.

But now that you've brought it up, it's got me digging further into my VPN setups to see if something looks out of whack there. I know I haven't changed VPNs or clients, but you never know. I'll do some digging and report back.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/docker/for-mac/issues/2820#issuecomment-434661264, or mute the thread https://github.com/notifications/unsubscribe-auth/AAr84dLd6wEjk8ms-narvjTRMVhQ_yJvks5uqZFigaJpZM4TaIiC .

[mightybs]

I just got it working again, but I'm not sure I can reproduce the failed state. At some point during my testing I did a factory reset, which still left some networking elements behind because I no longer had a "bip" entry in the daemon.json, but my containers were not using the default subnet, sadly I don't have the network inspect data from that time. Since then, I added "bip" back to the daemon.json, confirmed that the new network inspect output reflects the change, and now I can reach hosts on the VPN again!

I suspect that somewhere in all of the upgrading and troubleshooting, I had a slightly defunct network bridge. If I find that I'm having trouble again, I'll reach back out, but currently I'm back to working.

[drorata]

I believe I have the same issue even without docker-compose. I cannot access a resource from within a container but it works smoothly when accessing it directly from the host.

@zhyon404 I tried to change this bip configuration but it still didn't help. How did you decide what IP to use?

[drorata]

I used Rest to factory defaults and now it seems like I'm able to access the resources in the VPN.

[jpommerening]

@drorata thanks! I wasn't able to ping any host on the VPN network and your comment saved me from crawling (deeper) into the rabbit hole of networking :)

[cdenneen]

@drorata reset to factory defaults disables Kubernetes again. So yes you can ping hosts on VPN. When you re-enable Kubernetes it will conflict again with 10. VPN networks.

[jpommerening]

@cdenneen that's interesting. I had problems with 172. addresses though. Will confirm tomorrow whether enabling Kubernetes reintroduces the problem.

[jpommerening]

Update: Enabled k8s and can still connect to 172.x.x.x addresses.

[cdenneen]

Ok breaks with 10. VPN networks as Kubernetes uses similar. Have to modify and change to 10.x range that isn’t used by remote network. On Fri, Jan 4, 2019 at 6:13 AM Jonas notifications@github.com wrote:

Updates: Enabled k8s and can still connect to 172.x.x.x addresses.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/docker/for-mac/issues/2820#issuecomment-451416898, or mute the thread https://github.com/notifications/unsubscribe-auth/AAr84SRJh1h0ON5ZJfyEVCz8QMZbPbQkks5u_zdMgaJpZM4TaIiC .

[devn]

• Some suggest tunneling as a workaround
• Some say I should reset to factory defaults or reinstall.
• Others suggest using the extra_hosts stanza.
• Some say the order of connecting to the VPN and starting the docker daemon matters.
• The bip setting has been brought up.
• Finally, k8s being enabled or not has been discussed.

I do not have k8s enabled, have reset to factory defaults, tried the bip setting (tried setting to 192.168.1.5/24, and 172.17.107.0/24), tried changing the order of connecting to the VPN and starting docker. I do not have k8s enabled. My machine is at 192.168.1.8, running the OpenVPN client for mac connecting to 172.x.x.x. My linux friends don't have this issue.

I tried the extra_hosts stanza in my docker-compose file.

extra_hosts:
  - "db-host.foo.com:172.17.107.78"

On the host machine, my IP on en0 is 192.168.1.8. utun3 points at 172.27.235.171.

Any advice for troubleshooting further would be appreciated.

[devn]

While the work around is far from ideal, @mightybs solution of tunneling via ynetd worked for me. https://github.com/docker/for-mac/issues/2820#issuecomment-432528717

Also, thanks to Celmor on IRC for some additional troubleshooting help.

[DiegoGallegos4]

Just as an update and to save time debugging this issue, this issue's solution #1803; is no longer working and the docs for vpnkit are outdated.

[till]

This just happened to me today. Reset factory defaults worked — for whatever reason.

Docker version 18.09.2, build 6247962

[cdenneen]

Resetting to factory defaults disables Kubernetes. This is still issue is with Kubernetes networking conflicting with VPN networks of 10.0.0.0/8. So I’m my case I need to find a 10.x.0.0/24 to not conflict with my VPN networks at work.

[till]

Just for the record, I didn't have k8s enabled, so not sure how "related".

Also, Docker's IP range did not "conflict" with the VPN I was connected to. The only thing I enabled the other day was edge so I can --squash, but besides, my factory defaults are pretty default.

The biggest annoyance is having to log into docker hub and other custom registries now.

[tth80]

Adding another +1, had to reset docker to factory defaults. Didn't have kubernetes enabled.

Docker version 18.09.2, build 6247962

[pam-]

Another +1. The reset worked 🤷‍♀with Kubernetes enabled. Version 18.09.2, build 6247962

[kzvonov]

Had a similar VPN issue with the newest version of docker on my mac and nothing from above helped me.

The solution was to move to the previous one Hope this helps :)

[robertkern]

For what it's worth, I experience this problem on and off and doing a factory reset seems to fix it (but obviously is very annoying having to do this). I had upgraded to 2.1.0.1 hoping it might fix the issue but it didn't so downgraded to 2.0.0.2 (and did a factory reset). It's currently working (I'm currently able to access VPN network from containers going through the host computer VPN connection).

I don't customise any of the docker settings other than increasing the memory from 2GB to 4GB (so kubernetes is not enabled).

[kjenkins19]

Has anyone validated whether 2.1.0.2 fixes the issue yet, or found a more permanent solution? I try to avoid remaining behind versions.

[unstablectrl]

My VPN was also not working and I already had a different "bip" : "172.x.x.x/16" in daemon advanced options but before reseting try to delete docker networks docker network prune worked for me.

[yaizudamashii]

@kjenkins19 2.1.0.2 does not fix the issue. Doing factory reset didn't fix the problem described in the issue either. Doesn't work with 2.1.0.3 either.

[udalrich]

My VPN was also not working and I already had a different "bip" : "172.x.x.x/16" in daemon advanced options but before reseting try to delete docker networks docker network prune worked for me.

I do not have Kubernetes enabled, and docker has using a 192.x network while the vpn was at 172.x. Pruning networks removed several networks and fixed the problem for me.

[arg]

Not fixed in 2.1.0.5

Engine: 19.03.5
Compose: 1.24.1
Kubernetes: v1.14.8

can't access hosts in 172.x

[oberman]

I struggled with this issue for a little bit and eventually solved it. The root problem/solution for me is noted and solved in this thread (Docker using 172.17.x.x as a default for its networking and the bip setting to work around that). But it took me awhile as I didn't quite understand the bip setting at first. From what I can tell it's "bip":"CIDR" except rather than make all of the least significant bits zero in the left side of the CIDR you're supposed to pick your gateway IP. E.g. 172.30.0.0/16 is wrong and 172.30.0.1/16 is right. I additionally made the mistake of using /12 instead of /16 which maps all of 172.16.x.x through 172.31.x.x to docker0 (which obviously includes 172.17.x.x but is even worse...).

[Nklya]

Related https://github.com/docker/for-win/issues/1667

And still not fixed. You cannot change the Kubernetes network from GUI 😞

[renanwilliam]

I don't know if will help but my problem is I have many different projects running docker networks and for each one it is created a network on 172.24.0.0/16 CIDR.

My VPN is on 172.24.42.0/24 and one of this docker networks as overleaping with my VPN network. The solution for me was delete the overleaping network and create it again using other IP interval.

[jeffertan]

@renanwilliam How to check the VPN address? I have the same issues, tried above ways, it's not working when connecting under VPN.

[renanwilliam]

@jeffertan there many ways to do it like search in your VPN client, but one very simple is:

Connect to your VPN and execute the command in terminal: ifconfig | grep inet | grep mask

My system returns this output:

inet 127.0.0.1 netmask 0xff000000 
inet 10.40.1.163 netmask 0xffffff00 broadcast 10.40.1.255
inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255
inet 172.24.42.2 --> 172.24.42.2 netmask 0xffffff00 

First line is localhost, second is my local IP in LAN, third is docker network and the last is my VPN.

To check your docker networks, you can use this command:

docker network inspect `docker network ls -q` | grep Subnet

For me returns:

                    "Subnet": "172.21.0.0/16",
                    "Subnet": "172.20.0.0/16",
                    "Subnet": "172.17.0.0/16",
                    "Subnet": "172.19.0.0/16",
                    "Subnet": "172.18.0.0/16",
                    "Subnet": "172.23.0.0/16",
                    "Subnet": "172.22.0.0/16",

In my case, I will have the problem if exists a Docker network in 172.24.0.0/16 (probably the next created will be).

[blohinn]

Work for me: 1) Always restart docker daemon after connecting VPN on host. 2) Change bip in /etc/docker/daemon.json (for me work next config, bip is random, lol, but not confusing with vpn network) /etc/docker/daemon.json { "bip": "192.168.1.5/24" } 3) restart daemon, docker network prune and up containers again

sorry for my bad english((

[gonzamace]

After struggling for 3 days with this problem, it was indeed a crash problem between the sub-networks created by docker and the networks of my VPN.

The solution was not to change the version of Docker (18.09.2), but:

1. Add extra_hosts to my apache2 & workspace container in my docker-compose.yml:

   - "internal.server.company.com:172.162.4.1111"

2. Add in the daemon configuration (MacOS: taskbar > Preferences > Daemon > Advanced) the "default-address-pools" propertie:

   {
   "debug" : true,
   "default-address-pools" : [
   {
     "base" : "192.168.0.0/16",
     "size" : 24
   }
   ],
   "experimental" : false
   }

   Re-start daemon, docker network prune and up containers again

[causalbody]

+1 restoring to docker network factory fixed it for me

 ifconfig |grep inet |grep mask
    inet 127.0.0.1 netmask 0xff000000
    inet 192.168.0.x netmask 0xffffff00 broadcast 192.168.0.255
    inet 10.192.37.x --> 10.192.37.x netmask 0xffffff00

[zrml]

Docker Desktop for Mac v2.3.0.3; Docker Engine v19.03.8. I tried everything before resetting it. The only thing that did it was a full reset (it removed all images(!)) Now I can finally access the VPN. I think this is un-acceptable as a workaround as it is a network issue. I'd be OK with bringing down all running containers and resetting the network. Thanks

[cdenneen]

To all those who “reset”. Re-enable Kubernetes (reset disables) and see if you can still access 10.x.x.x VPN.

[5igm4]

My VPN was also not working and I already had a different "bip" : "172.x.x.x/16" in daemon advanced options but before reseting try to delete docker networks docker network prune worked for me. @unstablectrl

docker network prune resolved the issue for me without a hard reset. You're a life-saver!

[genii6]

I tried most of the suggestions in this thread and finally realised the root cause of the issue. Within my container, I was using curl to simulate an API call to an IP address which was behind a corporate VPN. For e.g. 172.18.30.180

After much tweaking around, I found that my issue was because of a subnet clash with the 172.18.xx.xx IP address. I listed out all my network bridges using docker network inspectdocker network ls -q``

and found that one of my network bridge's subnet was set to 172.18.0.0/16

I removed the offending user-defined bridge using docker network rm <id>

I was then able to connect to my VPN protected API. I undid all the suggested fixes in this thread including the daemon json bip config and the extra_hosts suggestion from docker-compose and the API was still able to connect.

I hope this helps someone and may explain why it seemed to work sometimes or it works after factory reset or after doing a network prune. The investigation was all done using Mac OS Catalina 10.15.6, Docker desktop CE 2.3.0.4 / Engine 19.03.12. You may of course use the bip setting approach to ensure that your default bridge never ever clashes with the API endpoint's IP you are trying to hit.

Cheers, Wayne

[ishakantony]

Hi, i faced this issue also. I did some troubleshooting using some suggestions from this thread, but i'm not sure which step solve my problem. What i did is:

1. Configure the JSON setting to add dns and dns-search using the vpn dns server.
2. Apply and restart
3. Do docker network prune
4. Remove the dns and dns-search
5. Apply and restart
6. Test curl and ping from inside container, and it works.

I did all those steps while connected to the vpn. Hope it helps.

[madrover]

I just faced this issue after upgrading my MacOS Docker to version 2.4.0.0

docker network prune

has been enough to solve it for me

[abdennour]

Best comment IMHO https://github.com/docker/for-mac/issues/2820#issuecomment-603218261 However, i will give more details :

diagnostic

1. What's your IP of VPN using GUI ?

in my case, it's 192.168.1.115

1. What's your IP of VPN using command line:

ifconfig | grep inet | grep mask

output:

inet 127.0.0.1 netmask 0xff000000 
inet 10.40.1.163 netmask 0xffffff00 broadcast 10.40.1.255
inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255
inet 192.168.1.115 --> 192.168.1.115 netmask 0xffffff00   <-------- this is VPN

2. Remove any docker network overlap with VPN network

# list docker network
docker network inspect `docker network ls -q` | jq 'map({Id, subnet: .IPAM.Config[0].Subnet})'

output:

[
  {
    "Id": "5f3a3eff213cc05b118e021a38fc2679a19f40f9a5bfa1b50dcce688d97fca86",
    "subnet": "172.29.0.0/16"
  },
  {
    "Id": "5bddbce06e52f9ebb1a63edd3881d68aa3b5e2ddce97452c9cd0b1b68c6fe4e7",
    "subnet": "192.168.112.0/20"
  },
  {
    "Id": "bd3cce93e1c75d97e90bf69e2a5e5bdc1c6b59e3da419d3ff0e2e105d76aeab0",
    "subnet": "172.18.0.0/16"
  },
  {
    "Id": "e3d4f2ef4267aa501656b720229f06de58eb79825d1e5a9a8caa1dd73f6d03f2",
    "subnet": "192.168.80.0/20"
  }
]

Remove all or at least remove subnets which overlap with VPN :

docker network rm ${Network_Id}

3. Use network which does not overlap with VPN

Because my VPN is 192.168.x.y, i chose 172.30.x.y

see solution below

solution

# docker-compose.yaml
version: '3.7'
services:
      my-service:
        image: something
        networks:
          my-network
            ipv4_address: 172.30.0.13
        # ....

networks:
  my-network:
    ipam:
      config:
       - subnet: 172.30.0.0/28
      # get ip list of subnet:  http://magic-cookie.co.uk/iplist.html

[VasiliyKozhevnikov]

@abdennour thanks it works

[mlecar]

My VPN was also not working and I already had a different "bip" : "172.x.x.x/16" in daemon advanced options but before reseting try to delete docker networks docker network prune worked for me. @unstablectrl

docker network prune resolved the issue for me without a hard reset. You're a life-saver!

docker network prune did the trick for me on mac. I was able to connect a service through VPN from container.

[KumarManoj-S]

Hi, I have tried the following,

• changing bip (my vpn ip is 10.6.x.y and I set the bip to 172.30.x.y).
• configuring the following,

  "default-address-pools" : [
  {
    "base" : "192.168.0.0/16",
    "size" : 24
  }
  ]

• Factory default reset
• docker network prune
• Downgrading the docker from 20.10.5 to 2.0.0.2

But nothing really helped in my case :(

I couldn't get the ping results when I'm connected to VPN using (docker run adiazmor/docker-ubuntu-with-ping ping 8.8.8.8). But it works the moment I turn off the VPN.

Any other solution to this?

Note: I'm on mac.