search

docker / for-mac

Bug reports for Docker Desktop for Mac
https://www.docker.com/products/docker#/mac
2.43k stars 117 forks source link

M1 Preview 7: crashes when on VPN (AnyConnect) #5208

Closed josmo closed 3 years ago

josmo commented 3 years ago

Expected behavior

Diagnostic logs

Diagnose succeeded

Steps to reproduce the behavior

  1. start anyconnect
  2. start docker
  3. crash
  4. disconnect from vpn
  5. start docker
  6. start containers
  7. start vpn
  8. cash

Seems to me that when I'm on the vpn the communication to linuxkit is lost. I've tried with the same macOS version and anyconnect version on intel and it works as expected. (I've reset, reinstalled etc)

stephen-turner commented 3 years ago

This is plausible, because with the new virtualisation framework on the M1 machines we had to use a different method of connecting to the VM. But we may not be able to fix it if AnyConnect is blocking the network path, it may require changes in your AnyConnect configuration.

josmo commented 3 years ago

Thanks @stephen-turner any information on what's changed to help hunt it down? Since Anyconnect is controlled at the corp level it might take a while to get that updated, although I image others will have similar issues, it might be nice to document what it takes to get the configuration to work :)

stephen-turner commented 3 years ago

We used to use sockets to connect to the VM, but that's currently not working in the new virtualisation framework so we now have to make a network connection. (@djs55 Please jump in if this is inaccurate). We are still talking to Apple about it, so we may be able to switch back at some point, which should result in fewer firewall problems.

o-mega commented 3 years ago

I had the same problem with Docker Preview 7 and pritunl vpn on my M1. Hope someone can find a solution to solve this issue 👀

krzaczek commented 3 years ago

HI,

I am also haveing issues with cisco anyconnect on M1 macbook pro but it doesn't crash my docker. I'm am able to connect/diconnect with docker enabled without any problems. The only problem I have is that the docker containers can't access VPN networks (vpn passthrough). This seems to work fine on my Intel based macbook pro.

josmo commented 3 years ago

Thanks for the info @stephen-turner that totally makes sense. Would love for the direct socket to work since it would it would make things like this a non issue, hopefully you'll come up with something with Apple.

@o-mega I was able to workaround temporarily with openconnect since it seems to setup routing order in a better way (there's probably a way to script updating the routes to allow the connection but I haven't looked at that yet)

@krzaczek I bet it's because your anyconnect is setup for split tunneling and not routing the linuxkit interface over it. I've tried openconnect and it works on split tunneling as well using vpn-slice but there's probably also a way to fix the routing with anyconnect I just didn't get that far when openconnect was working.

o-mega commented 3 years ago

@josmo how did you connected to ovpn? I'm not able to ping remote server via ovpn inside the container, but same time everything is ok with ping from the local console.

whydna commented 3 years ago

@stephen-turner I'm having the same issue.

Connected to VPN with OpenVPN client on host machine. Everything works from local terminal, but within the container its not able to access the network.

@o-mega did you find a solution?

o-mega commented 3 years ago

@whydna unfortunately no 😔 I've tried to use proxy with same result. Also I tried to share a hotspot from my phone connected to VPN, but got the same results: the local terminal can see remote host, but no connection inside containers.

At the moment, I have to use my old intel based macbook to work with docker 😣

josmo commented 3 years ago

@josmo how did you connected to ovpn? I'm not able to ping remote server via ovpn inside the container, but same time everything is ok with ping from the local console.

@o-mega I didn't do anything special :/ everything just worked for me and openconnect. You will need to make sure the vpnc-script is correct (that's what setups up all the routes and what not). I use a combo of the default one (for routing all traffic) and one based off vpn-slice (for split routing) I haven't had any problems with either.

o-mega commented 3 years ago

I found a temporary solution for myself, while waiting for the official upgrade from docker team (I think it wouldn’t before golang for m1 releases).

I took my old router Linksys EA4500 and updated it with OpenWrt firmware. This trick allowed me to enable Open VPN connection through the router itself.

As the result, all vpn connections are manages not by osx. And the Docker Preview also see only one network. It is not as secure as it should, because vpn shares for all router connections. And I have to use two routers: one for guests, and one for my job. But, anyway, now everything looks fine and I can do my job at least.

yvess commented 3 years ago

"openconnect" as workaround worked also for me. With brew install openconnect && brew install openconnect-gui it's easy to use with the gui application.

Jeuners commented 3 years ago

I use Tunnelblick VPN with the Apple M1. There is no crash, but Docker does not pass data through the VPN, so no internet connection is possible.

marcosanson commented 3 years ago

@Jeuners I have the same configuration and I can see the same issue. I tried to use tcpdump and look at packet from osx. I see container ip trying to connect over the vpn. I think Tunnelblick does not permit this tcp connections. You or someone knows if docker has some configuration to masq/nat container ip with host ip? I think this could be enough.

wgebis commented 3 years ago

Hi, I'm facing the same issue on MacBook M1

$ docker pull docker/getting-started
Using default tag: latest
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Looks that it is also related to lack of internet connection by Docker daemon. Without VPN it works, as well as x86 machine with the same configuration.

jostack commented 3 years ago

I had the same problem with Docker Preview 7 and Pulse Secure on my M1.

Can´t connect to remote database and can´t make a pull imagen over VPN ..... Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) ....too.

yvess commented 3 years ago

"openconnect" as workaround worked also for me. With brew install openconnect && brew install openconnect-gui it's easy to use with the gui application.

The native macOS vpn client also supports Cisco IPSec, I'm using that know.

wgebis commented 3 years ago

Here s a workaround for NAT issues after VPN connection: https://github.com/lastcosmonaut/docker-m1-vpn-workaround It works for me (OpenVPN).

MarcoGarofalo94 commented 3 years ago

I use Forticlient as vpn client. Same issue, no connection from the container.

PanMisza commented 3 years ago

I use Forticlient as vpn client. Same issue, no connection from the container.

@MarcoGarofalo94 To resolve issue with FortiClient, use openfortivpn with set-routes and pppd-use-peerdns enabled - works for me perfectly.

MarcoGarofalo94 commented 3 years ago

I use Forticlient as vpn client. Same issue, no connection from the container.

@MarcoGarofalo94 To resolve issue with FortiClient, use openfortivpn with set-routes and pppd-use-peerdns enabled - works for me perfectly.

Thank you very much! Now my container can reach the VPN, unfortunately my host loses connection to internet even with --set-routes=1

Jeuners commented 3 years ago

Here s a workaround for NAT issues after VPN connection: https://github.com/lastcosmonaut/docker-m1-vpn-workaround It works for me (OpenVPN).

works fine!

0reoo commented 3 years ago

Here s a workaround for NAT issues after VPN connection: https://github.com/lastcosmonaut/docker-m1-vpn-workaround It works for me (OpenVPN).

works fine!

我用的Cisco，貌似还是不行

whydna commented 3 years ago

Here s a workaround for NAT issues after VPN connection: https://github.com/lastcosmonaut/docker-m1-vpn-workaround It works for me (OpenVPN).

This did help a bit, but I'm still getting connection errors from Django running inside the container when it tries to connect to a mysql instance behind the firewall/vpn.

@wgebis Is the fix above limited only to pulling 3rd party docker repos?

marcosanson commented 3 years ago

Here s a workaround for NAT issues after VPN connection: https://github.com/lastcosmonaut/docker-m1-vpn-workaround It works for me (OpenVPN).

works fine!

wgebis commented 3 years ago

@whydna I believe that the fix solves the NAT issue - without it docker daemon is not able to determine tun device, that's why daemon is not able to make necessary rules related to NAT. Without that fix docker daemon is not able to make any call to remote resources via tun device (VPN) - so you observe no internet access.

fnaticshy commented 3 years ago

Here s a workaround for NAT issues after VPN connection: https://github.com/lastcosmonaut/docker-m1-vpn-workaround It works for me (OpenVPN).

its work!

maxbrizolla commented 3 years ago

Is not work for me,

I use openfortivpn and tried to use --pppd-no-peerdns --set-routes=1, but, not work. And, tried to use https://github.com/lastcosmonaut/docker-m1-vpn-workaround and not work yet.

docker pull nexus.mycompany.com/java8openjdk:latest
Error response from daemon: Get https://nexus.mycompany.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Mac M1 Docker V 3.1.0 (60984)

DimasGon commented 3 years ago

I noticed that my Docker Preview working fine when I am connected to corp VPN (AnyConnect), but when I updated Docker Preview to 3.1.0 (60984) it started crashing... Docker Preview was updated today (February, 18)

@stephen-turner can you, please, help me? Which version of Docker Preview I could have used when I downloaded it on February, 4? Everything worked fine in that version. Can't find any versions history :(

Thanks a lot!

stephen-turner commented 3 years ago

There have only been two public previews, one in December and one last week.

jca41 commented 3 years ago

I noticed that my Docker Preview working fine when I am connected to corp VPN (AnyConnect), but when I updated Docker Preview to 3.1.0 (60984) it started crashing... Docker Preview was updated today (February, 18)

@stephen-turner can you, please, help me? Which version of Docker Preview I could have used when I downloaded it on February, 4? Everything worked fine in that version. Can't find any versions history :(

Thanks a lot!

Maybe this? https://desktop.docker.com/mac/stable/arm64/60902/Docker.dmg

stephen-turner commented 3 years ago

No, 60902 is basically the same as 60984 but with a bug that we later fixed.

The old preview is no longer available. It was time-limited anyway so would have stopped working.

davemikenorton commented 3 years ago

https://github.com/docker/for-mac/issues/5208#issuecomment-781472885

I have the same problem with Nexus.

PiotrRaszkowski commented 3 years ago

Hi, I have the same problem. If Docker is started, once you connect to the vpn (in my case AnyConnect) it crashes.

Error from Docker Desktop: 2021-02-22T16:55:37Z dockerd time="2021-02-22T16:55:37.363437959Z" level=error msg="Handler for GET /v1.24/services returned error: This node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again."

PiotrRaszkowski commented 3 years ago

It seams that openconnect + openconnect-gui as a workaround for AnyConnect works fine!

othmaan commented 3 years ago

Issue was solved when I disabled macOS firewall & closed AnyConnect VPN client. Once Docker starts everything works fine & I can enable the firewall..

built2order commented 3 years ago

Issue was solved when I disabled macOS firewall & closed AnyConnect VPN client. Once Docker starts everything works fine & I can enable the firewall..

What's the configuration of your firewall? Is it configured for specific apps or automatically allow signed software to receive incoming connections?

thecosmicfrog commented 3 years ago

Confirm that this is still an issue for me in Docker Desktop M1 Preview RC1 (2021-03-18). Docker Desktop will not start when connected to Cisco AnyConnect VPN.

stephen-turner commented 3 years ago

Right, that's exactly as we documented.

thecosmicfrog commented 3 years ago

Thanks @stephen-turner. Is it expected that this will be fixed prior to the stable release? I don't have a choice of VPN clients at work unfortunately.

stephen-turner commented 3 years ago

24 hours ago I would have said no ... but we've just made an architectural change that might fix it.

Everyone watching this thread, please try the new build at https://docs.docker.com/docker-for-mac/apple-m1/ and give us feedback whether it solves your VPN problems (and whether it causes any new problems!). Thank you.

lucashenriquesd commented 3 years ago

I just tried the new version(RC2) and it fixed my VPN problem with globalprotect. I also reported to https://github.com/lastcosmonaut/docker-m1-vpn-workaround that the workaround was indeed not needed in RC2 anymore.

thecosmicfrog commented 3 years ago

Already commented in #5322, but commenting again here for visibility:

@stephen-turner RC2 is looking good for me so far! Uninstalled RC1, killed any lingering processes and installed RC2 (all while connected through Cisco AnyConnect VPN client). Docker for Mac starts up and I can docker login, docker run, docker pull, etc.

What a fantastic way to close out a week! Thank you for getting this fixed - it's a massive unblocker for me.

I'll update the other issue if I run into any problems.

mbatesco commented 3 years ago

RC2 is looking good for me so far as well. I used the Check update process rather than reinstall and it worked fine. I have gone through multiple reboots Docker comes up fine with both the firewall and VPN up and running.

Thanks this was a big step forward.

davidbarton commented 3 years ago

RC2 fixed the VPN for me, thank you

tangledhelix commented 3 years ago

RC2 is working with Anyconnect VPN for me now. Previously I could not use docker login, pull, push to either Docker Hub or to my private container registry in Gitlab. Thanks!

simoncpu commented 3 years ago

I've experienced this problem ever since I downloaded the M1 Preview and I've been patiently waiting for a fix. I'm downloading RC3 to now!

azaeng04 commented 3 years ago

I am experiencing a similar error on the RC3 build. I get this 504 Gateway error when proxying requests sent from localhost to an external server.

The error I receive:

Error occured while trying to proxy to: localhost:4200/account/me/?format=json

Screenshot 2021-04-02 at 11 00 23

Screenshot 2021-04-02 at 10 55 06

Help would be much appreciated

FortiClient VPN v6.4.2.1305 for macos Docker image used: node:15.13.0-alpine3.13 Angular v9.1.0

Please let me know if there is any other information required.

djs55 commented 3 years ago

@azaeng04 thanks for the report. Could you upload diagnostics from the whale menu -> troubleshoot -> get support and quote the ID here? Is there any chance you could make a small repro example or maybe show me the full HTTP request that failed (redacting specific names, cookies etc is fine-- I just need to know enough to repro it locally with a test server). You mention proxying requests from localhost -- is this the host localhost or the VM? Do you have macOS system proxies setup or have you configured proxies from the docker desktop UI?

azaeng04 commented 3 years ago

@djs55 thanks so much for the speedy response. Diagnostics indicates that it has succeeded.

Screenshot 2021-04-02 at 11 21 43

It is an Angular v9.1.0 based project with a proxy config set up:

const PROXY_CONFIG = [
    {
        context: [
            "/login",
            "/account",
            "/logout",
            "/api",
            "/noninteractivelogin",
        ],
        target: `https://${host}`,
        secure: false,
        changeOrigin: true,
        logLevel: "debug",
        headers: { referer: `https://${host}/portal/` },
    },
];

The host variable is some arbitrary server that acts as the poxy: e.g. 10.x.x.x

The command I am executing is ng serve -- --host 0.0.0.0 -c proxy

I can try to create a repository that tries to replicate this error.

Kind regards