search

alexellis / k3sup

bootstrap K3s over SSH in < 60s 🚀
https://github.com/sponsors/alexellis
Other
6.3k stars 376 forks source link

node-ip is not set to external ip #136

Closed Hades32 closed 4 years ago

Hades32 commented 4 years ago

When creating a server with k3sup v0.9.1 you have to provide the public IP. But the IP being announced to other nodes on joining is just that of the first network interface. As the server doesn't know it's own public IP (my hoster is using some DNAT). This leads to the agents not being able to fully join.

Expected Behaviour

k3s server --node-ip=THE_IP should be set in the systemd file for k3s

Current Behaviour

parameter is not set and seems to be inferred from the available network interfaces.

Possible Solution

Just pass the parameter along. At least if the given IP is not found on the server.

Steps to Reproduce (for bugs)

  1. create a server behind a NAT
  2. try to join as agent from behind another NAT

Context

My cloud provider is using DNAT (IP is stable but could be moved to another host easily) and I wanted to join my Pine64 into that cluster and my home network is NATed, too. (IP is not stable of that one)

Your Environment

alexellis commented 4 years ago

Hi, what error did you receive and how did you test the proposed fix?

alexellis commented 4 years ago

It also sounds like you need to use the wireguard VPN, since by default all nodes need to have visibility of one another. I have tried the scenario you mention, I got it to connect and list the two nodes, but pod networking didn't work due to an open issue with k3s.

Hades32 commented 4 years ago

Journalctl of k3s showed, connecting to (wrong up) indefinitely. Before the fix and then it worked.

But unfortunately you are right. Now pod networking isn't working. Didn't realize that yesterday.

I thought that I wouldn't need a VPN, because of k3s' kube-proxy tool and because I have an overlay network with flannel...?

Hades32 commented 4 years ago

Oh, and Merry Christmas to you Alex :) 🎄

alexellis commented 4 years ago

I thought the same, but after looking into the docs, flannel assumes pod to pod reachability, that's why we need to enable the wireguard support. There is a flag for that already.

Could you send a patch that sets the external node IP for whatever the --ip flag was? I don't think a new flag is needed.

Happy Christmas to you too, please consider supporting my work for 2020 if you don't already with GitHub Sponsors :evergreen_tree:

Ehco1996 commented 4 years ago

i found this too，my k3s server is behind the NAT and the eth0's inet is " 172.17.32.236" which is the inner private ip

when i join another node form another VPS，agent log is

k3s[20942]: time="2020-01-01T13:20:54.107720280Z" level=error msg="Failed to connect to proxy" error="dial tcp 172.17.32.236:6443: connect: connection timed out"

it use master node private ip not the Public IP

alexellis commented 4 years ago

@Ehco1996 why don't you try running the commands with the public IPs?

Ehco1996 commented 4 years ago

@alexellis i have already use the public ip , i found this https://github.com/rancher/k3s/issues/727 , maybe i need use some VPN like wireguard to linked muti node behind NAT

alexellis commented 4 years ago

Yes you do. We have a flag for it already, use that?

Ehco1996 commented 4 years ago

@alexellis thanks reply , your mean --ipsec? i will try!

after one day search i found this: here i found after change the node Annotations flannel.alpha.coreos.com/public-ip to the real public ip, everything is ok!