Docker-machine not working when connected to VPN

[istian]

docker-machine just won't work when connected to VPN, I'm on windows 7 and connected to corporate VPN.

Error checking TLS connection: Error checking and/or regenerating the certs: There was an error validating certificates
 for host "xxx.xxx.99.103:2376": dial tcp xxx.xxx.99.103:2376: i/o timeout
You can attempt to regenerate them using 'docker-machine regenerate-certs [name]'.
Be advised that this will trigger a Docker daemon restart which might stop running containers.

tried this https://www.iancollington.com/docker-and-cisco-anyconnect-vpn/

and this

myhost=default
myip=`docker-machine ip $myhost`
docker-machine stop $myhost
vboxmanage modifyvm "$myhost" --natpf1 "docker,tcp,$myip,2376,,2376"
docker-machine start $myhost

Is there a workaround, please? If not, can someone confirm that this isn't possible?

[jonases]

I have the same issue on mac, and I followed https://www.iancollington.com/docker-and-cisco-anyconnect-vpn/ and that resolved it hopefully that helps

[jamshid]

Whew this a subtle, hard to diagnose problem. Thanks @jonases that led me to figure out why my remote docker access was failing. I never would have guessed this was caused by a recent Viscosity VPN settings change.

# Curl seemed to work fine, this 404 is expected, validates connectivity and certs
$ curl -i --cert ~/.docker/machine/certs/cert.pem --key ~/.docker/machine/certs/key.pem --cacert ~/.docker/machine/certs/ca.pem https://mydocker.example.com:2376
HTTP/1.1 404 Not Found
Content-Type: application/json
Date: Wed, 28 Nov 2018 01:20:24 GMT
Content-Length: 29

{"message":"page not found"}

# WTF does this mean
$ docker-machine env mydocker.example.com
Error checking TLS connection: Host is not running

# Eh --debug is not much help
$ docker-machine --debug env mydocker.example.com
Docker Machine Version:  0.16.0, build 702c267f
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver generic
Plugin server listening at address 127.0.0.1:64633
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(mydocker.example.com) Calling .GetURL
Error checking TLS connection: Host is not running

The problem was fixed by changing Viscosity => edit Connection => Network tab => DNS Setting => must be `Full DNS (Use VPN DNS for all traffic), not Automatic or Split.

$ docker-machine --debug env mydocker.example.com
Docker Machine Version:  0.16.0, build 702c267f
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver generic
Plugin server listening at address 127.0.0.1:65424
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(mydocker.example.com) Calling .GetURL
Reading CA certificate from /Users/jamshid/.docker/machine/certs/ca.pem
Reading client certificate from /Users/jamshid/.docker/machine/certs/cert.pem
Reading client key from /Users/jamshid/.docker/machine/certs/key.pem
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://mydocker.example.com:2376"
export DOCKER_CERT_PATH="/Users/jamshid/.docker/machine/machines/mydocker.example.com"
export DOCKER_MACHINE_NAME="mydocker.example.com"
# Run this command to configure your shell: 
# eval $(docker-machine --debug env mydocker.example.com)
Making call to close driver server
(mydocker.example.com) Calling .Close
Successfully made call to close driver server
Making call to close connection to plugin binary

[leisenstein]

We just resolved this issue at my company. The issue was with our Cisco Anypoint Connect configuration.

By default, DockerNAT creates a Virtual network to communicate over and it is set to 10.0.75.X. Even though everything is local, network communications are sent over this virtual network.

The problem was our Anypoint Connect was routing ALL traffic to another office before sending it out. So, instead of traffic going to my Docker instance, it was being routed to an office in another state.

We had to modify the Anypoint profile to skip routing on the 10.0.75.X network so that traffic would not get re-routed. After that, everything was working.

[szamaniWP]

@leisenstein Have same problem, can you write where I can change routing in anypoint? Can it be changed in client?

[answerquest]

@leisenstein is there anything a company's employee can do at their end? Example: Here they talk about changing nameserver, located in /etc/resolv.conf after ssh'ing into the VM. So, how about I use a nameserver that my VPN is using?

[jnthnwn]

I was running into the same problem. Aside from rebooting, the only thing that has worked to solve this issue for me is from https://github.com/onejli/docker-vpn-helper, notably this excerpt:

A better solution than manually re-adding routes is to help VirtualBox fix the routing table. The Cisco AnyConnect VPN agent removes/redirects routes upon connection, but doesn't restore them after disconnecting. This seems to make the VirtualBox network kernel modules very unhappy. After dropping off of VPN, VirtualBox is able to add host-only network adapters, but it is NOT able to add the routes needed to connect them. This thread describes one solution.

1. Disconnect from the VPN (you can actually leave the Cisco AnyConnect application/services running)
2. Stop all VirtualBox processes (i.e., all VMs and the GUI)
3. Restart the VirtualBox kernel modules sudo /Library/Application\ Support/VirtualBox/LaunchDaemons/VirtualBoxStartup.sh restart