search

rancher-sandbox / rancher-desktop

Container Management and Kubernetes on the Desktop
https://rancherdesktop.io
Apache License 2.0
6.02k stars 285 forks source link

Unable to connect to host VPN through rancher desktop on Macos #2776

Open yevon opened 2 years ago

yevon commented 2 years ago

Actual Behavior

When you have a VPN connected in the host, kubernetes nodes are unable to communicate with servers in the VPN. This doesn't happen in windows, this works nicely. Related issue in Lima VM repository:

https://github.com/lima-vm/lima/issues/587

Steps to Reproduce

Connect a VPN on macOS in the host, and try to ping from one of the nodes of the cluster to a computer in the host VPN, it will say "unreachable".

Result

Unreachable hosts within kubernetes hoster via VPN in the host.

Expected Behavior

It should be able to communicate with any computer within the cluster that the host has access to. If I make changes to underlying Lime VM routing tables, those changes could be lost if I update rancher os.

Additional Information

No response

Rancher Desktop Version

1.5.1

Rancher Desktop K8s Version

1.21

Which container engine are you using?

containerd (nerdctl)

What operating system are you using?

macOS

Operating System / Build Version

macOs Monterey 12.0.1

What CPU architecture are you using?

arm64 (Apple Silicon)

Linux only: what package format did you use to install Rancher Desktop?

No response

Windows User Only

No response

adamkpickering commented 2 years ago

This issue started in a discussion: https://github.com/rancher-sandbox/rancher-desktop/discussions/2740. I should note that @yevon was using wireguard as their VPN.

chriscasola commented 2 years ago

I am having the same issue, using GlobalProtect VPN.

Edit: adding some more detail

This seems to be because certain hosts are being routed to the network interfaces created by docker for local docker networks created using docker network create.

If I delete my docker networks this issue is resolved.

jandubois commented 2 years ago

Thanks @chriscasola !

@yevon I've just tested the setup you described: I've connected to a remote VPN using Viscosity. Then I've deployed a container on the Rancher Desktop kube cluster.

I started an interactive session inside that container and verified that the name of a remote host resolves. That proves that DNS lookup follows the split-DNS configuration provided by Viscosity.

Then I installed openssh into the container and started an ssh session to the remote machine on the other side of the VPN, and that worked too, showing that packets where routed correctly. It felt a bit slow, but was otherwise working fine.

So I cannot reproduce the problem you are having. Can you provide additional details? Otherwise I don't know what else we can do.

yevon commented 2 years ago

Thanks @chriscasola !

@yevon I've just tested the setup you described: I've connected to a remote VPN using Viscosity. Then I've deployed a container on the Rancher Desktop kube cluster.

I started an interactive session inside that container and verified that the name of a remote host resolves. That proves that DNS lookup follows the split-DNS configuration provided by Viscosity.

Then I installed openssh into the container and started an ssh session to the remote machine on the other side of the VPN, and that worked too, showing that packets where routed correctly. It felt a bit slow, but was otherwise working fine.

So I cannot reproduce the problem you are having. Can you provide additional details? Otherwise I don't know what else we can do.

Hi thanks for testing this! Might be vpn related then, Any special config? Might be due to allowed subnetworks ip mask in the vpn?. I will try to reach the user with the mac for further testing. I will try what @chriscasola suggests also.

yevon commented 2 years ago

Did you activate IP forwarding or set up some nat routes?

jandubois commented 2 years ago

Did you activate IP forwarding or set up some nat routes?

No, I just connected via Viscosity with my OpenVPN profile, and that was it.

chriscasola commented 2 years ago

@jandubois should I file a separate issue for the docker network problem we're having at my company? Docker networks seem to work fine for days/weeks but then all of a sudden none of the containers can reach hosts on the VPN. The only solution is to delete all docker networks and containers and recreate them.

It seems like a routing issue, where connections from within the containers start routing to the local network instead of the VPN network, but I haven't been able to confirm that. Any tips on how to debug would be appreciated.

chriscasola commented 2 years ago

Bumping this again because it's becoming really frustrating to have to delete all my docker networks and containers and recreate them to resolve this issue.

Is there anything I can do to help move this along?

jandubois commented 2 years ago

should I file a separate issue for the docker network problem we're having at my company? Docker networks seem to work fine for days/weeks but then all of a sudden none of the containers can reach hosts on the VPN. The only solution is to delete all docker networks and containers and recreate them.

Yes, please file a separate issue, as that sounds like a different problem.

However, I'm not sure what we can do about it unless we can reproduce the problem.

So restarting Rancher Desktop or even rebooting the host machine does not resolve the problem? You have to delete the networks and containers?

@Nino-K Do you have any ideas?

Nino-K commented 2 years ago

Docker networks seem to work fine for days/weeks but then all of a sudden none of the containers can reach hosts on the VPN. The only solution is to delete all docker networks and containers and recreate them.

@chriscasola when the issue occurs, have you tried inspecting the subnet IP address range that is used by the docker network? to make sure it is not conflicting with the VPN network?

yevon commented 2 years ago

Seems that docker desktop faced sames issues with mac m1 and big sur, https://github.com/docker/for-mac/issues/5322 @jandubois , is your mac an M1 with big sur? I will try some of the workarround they mention on this issue.

jandubois commented 2 years ago

is your mac an M1 with big sur?

No, it is an Intel machine with Catalina. My M1 machine with Big Sur is on the other side of the VPN...

chriscasola commented 2 years ago

Spun off my issue to #3161 although I'm not convinced these are actually different issues.

chriscasola commented 2 years ago

@Nino-K I think you were right about the docker network subnets conflicting with the VPN network. I found this issue in moby while digging around and it seems like I can change the default subnets for docker network create which should solve my issue, will report back.