Open vladimir22 opened 3 years ago
Maybe it is not related, but some things seem odd to me.
Your nodes have private IPs and you are using mesh-granularity=local
, but still the nodes are WireGuard peers of each other. Can you share the labels of your nodes? Maybe you put them in different locations even though they are in the same private network? This is also suggested by the graph you shared.
I am not sure if it can work to have nodes in the same "real" location, but in different "Kilo" locations.
EDIT: If you still want to encrypt all traffic, use mesh-granularity=full
instead of the different locations.
Hmm the funny thing here is that your private IPs are being used as the public endpoints for WireGuard. This creates some tricky situations. The problem here (i think) is that the master node is dropping martian packets. Imagine the following situation:
172.25.132.0/23 dev eth0 proto kernel scope
);-A KILO-NAT -d 172.25.132.35/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
);10.44.1.0/24 via 10.4.0.3 dev kilo0 proto static
);This is all a funny side effect of the reuse of the private IPs in the cluster as public endpoints. In order for this to work, node1 would have to send packets to master's IP address over the WireGuard network, but we can't do this because this IP address is the tunnel's endpoint. The only way around this is to ensure that IP addresses are ONLY either endpoints OR private IPs but not both. Questions for you:
I wonder if there's a nicer way we could deal with this in Kilo to enable fully private clusters @leonnicolas
Also, can you share the logs from the Kilo pod on master? Ideally with debug log level :))
@squat I appreciate your detailed response, yes you right, we are using:
The idea was to cover all internal RKE nodes by VPN connections + add custom external peers.
The configuration above related to the next location settings:
kubectl label node foundation-musanin-master kilo.squat.ai/location="foundation-musanin-master" --overwrite
kubectl label node foundation-musanin-node-1 kilo.squat.ai/location="foundation-musanin-node-1" --overwrite
kubectl label node foundation-musanin-node-2 kilo.squat.ai/location="foundation-musanin-node-2" --overwrite
In that case, I was able to see already working VPN connections between the nodes, and only one minor thing ( kube-api access from non-master POD) spoiled all the stuff :)
Let me share another bunch of data for other cases:
Service discovery is not working, I CANNOT access to any POD because sudo wg
is not configured properly
, details below:
kubectl label node foundation-musanin-master kilo.squat.ai/location="musanin" --overwrite
kubectl label node foundation-musanin-node-1 kilo.squat.ai/location="musanin" --overwrite
kubectl label node foundation-musanin-node-2 kilo.squat.ai/location="musanin" --overwrite
kgctl graph
digraph kilo {
label="10.4.0.0/16";
labelloc=t;
outputorder=nodesfirst;
overlap=false;
"foundation-musanin-master"->"foundation-musanin-node-1"[ dir=both ];
"foundation-musanin-master"->"foundation-musanin-node-2"[ dir=both ];
subgraph "cluster_location_location:" {
label="location:";
style="dashed,rounded";
"foundation-musanin-master" [ label="location:\nfoundation-musanin-master\n10.44.0.0/24\n172.25.132.35\n10.4.0.1\n172.25.132.35:51820", rank=1, shape=ellipse ];
"foundation-musanin-node-1" [ label="location:\nfoundation-musanin-node-1\n10.44.1.0/24\n172.25.132.55", shape=ellipse ];
"foundation-musanin-node-2" [ label="location:\nfoundation-musanin-node-2\n10.44.2.0/24\n172.25.132.230", shape=ellipse ];
}
;
subgraph "cluster_peers" {
label="peers";
style="dashed,rounded";
}
;
}
kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default master 1/1 Running 0 13m 10.44.0.4 foundation-musanin-master <none> <none>
default node1 1/1 Running 0 13m 10.44.1.4 foundation-musanin-node-1 <none> <none>
default node2 1/1 Running 0 13m 10.44.2.5 foundation-musanin-node-2 <none> <none>
kube-system coredns-7c5566588d-6zsf8 1/1 Running 0 136m 10.44.1.2 foundation-musanin-node-1 <none> <none>
kube-system coredns-7c5566588d-jsf5t 1/1 Running 0 136m 10.44.0.2 foundation-musanin-master <none> <none>
kube-system coredns-autoscaler-65bfc8d47d-h525p 1/1 Running 0 136m 10.44.2.2 foundation-musanin-node-2 <none> <none>
kube-system kilo-rm7xl 1/1 Running 0 3m43s 172.25.132.230 foundation-musanin-node-2 <none> <none>
kube-system kilo-tb5v9 1/1 Running 0 3m43s 172.25.132.35 foundation-musanin-master <none> <none>
kube-system kilo-tvgpl 1/1 Running 0 3m43s 172.25.132.55 foundation-musanin-node-1 <none> <none>
kube-system metrics-server-6b55c64f86-6t49b 1/1 Running 0 136m 10.44.2.3 foundation-musanin-node-2 <none> <none>
kube-system rke-coredns-addon-deploy-job-hb6tp 0/1 Completed 0 136m 172.25.132.35 foundation-musanin-master <none> <none>
kube-system rke-ingress-controller-deploy-job-lbxn6 0/1 Completed 0 136m 172.25.132.35 foundation-musanin-master <none> <none>
kube-system rke-metrics-addon-deploy-job-tv2b8 0/1 Completed 0 136m 172.25.132.35 foundation-musanin-master <none> <none>
kube-system rke-network-plugin-deploy-job-tlchx 0/1 Completed 0 136m 172.25.132.35 foundation-musanin-master <none> <none>
local-path-storage local-path-provisioner-5bd6f65fdf-525fc 1/1 Running 0 133m 10.44.0.3 foundation-musanin-master <none> <none>
pf echoserver-977db48cd-fcvsj 1/1 Running 0 15m 10.44.2.4 foundation-musanin-node-2 <none> <none>
kubectl get service -n pf -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
echoserver ClusterIP 10.45.241.43 <none> 8080/TCP 98m app=echoserver
## failed access by hostname
kubectl exec -it master -- curl -kv http://echoserver.pf:8080
kubectl exec -it node1 -- curl -kv http://echoserver.pf:8080
kubectl exec -it node2 -- curl -kv http://echoserver.pf:8080
* Could not resolve host: echoserver.pf
* Closing connection 0
curl: (6) Could not resolve host: echoserver.pf
command terminated with exit code 6
## failed access by service IP
kubectl exec -it master -- curl -kv http://10.45.241.43:8080
kubectl exec -it node1 -- curl -kv http://10.45.241.43:8080
kubectl exec -it node2 -- curl -kv http://10.45.241.43:8080
* Trying 10.45.241.43:8080...
* connect to 10.45.241.43 port 8080 failed: Host is unreachable
* Failed to connect to 10.45.241.43 port 8080: Host is unreachable
* Closing connection 0
curl: (7) Failed to connect to 10.45.241.43 port 8080: Host is unreachable
command terminated with exit code 7
## only POD on the same node has acces by POD IP
kubectl exec -it master -- curl -kv http://10.44.2.4:8080
kubectl exec -it node1 -- curl -kv http://10.44.2.4:8080
* Trying 10.44.2.4:8080...
* connect to 10.44.2.4 port 8080 failed: Host is unreachable
* Failed to connect to 10.44.2.4 port 8080: Host is unreachable
* Closing connection 0
curl: (7) Failed to connect to 10.44.2.4 port 8080: Host is unreachable
command terminated with exit code 7
kubectl exec -it node2 -- curl -kv http://10.44.2.4:8080 - OK
foundation-musanin-master configuration:
sudo wg
interface: kilo0
public key: c7yyiSaA9nvLVFz60Rkr42+xdvC4BVPaGDKJ+5v5QTU=
private key: (hidden)
listening port: 51820
sudo ip r
default via 172.25.132.1 dev eth0 proto dhcp metric 100
10.4.0.0/16 dev kilo0 proto kernel scope link src 10.4.0.1
10.44.0.0/24 dev cni0 proto kernel scope link src 10.44.0.1
10.44.0.0/24 dev kube-bridge proto kernel scope link src 10.44.0.1
10.44.1.0/24 via 172.25.132.55 dev tunl0 proto static onlink
10.44.2.0/24 via 172.25.132.230 dev tunl0 proto static onlink
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.25.132.0/23 dev eth0 proto kernel scope link src 172.25.132.35 metric 100
sudo ip a
43: kilo0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
link/none
inet 10.4.0.1/16 brd 10.4.255.255 scope global kilo0
valid_lft forever preferred_lft forever
foundation-musanin-node-1 configuration:
sudo wg
interface: kilo0
public key: yA6LdCuJT7y+pRNvlhds8GeeEGoT1Q/PUhF++GZ8gB0=
private key: (hidden)
listening port: 51820
sudo ip r
default via 172.25.132.1 dev eth0 proto dhcp metric 100
10.44.1.0/24 dev cni0 proto kernel scope link src 10.44.1.1
10.44.1.0/24 dev kube-bridge proto kernel scope link src 10.44.1.1
10.44.2.0/24 via 172.25.132.230 dev tunl0 proto static onlink
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.25.132.0/23 dev eth0 proto kernel scope link src 172.25.132.55 metric 100
sudo ip a
27: kilo0: <POINTOPOINT,NOARP> mtu 1420 qdisc noqueue state DOWN group default qlen 1000
link/none
inet 10.4.0.1/16 brd 10.4.255.255 scope global kilo0
valid_lft forever preferred_lft forever
foundation-musanin-node-2 configuration:
sudo wg
interface: kilo0
public key: IErj++lf80jkWOEEVsH97G6tTbNGViCZ12s2Gedl5kg=
private key: (hidden)
listening port: 51820
sudo ip r
default via 172.25.132.1 dev eth0 proto dhcp metric 100
10.4.0.1 via 172.25.132.35 dev tunl0 proto static onlink
10.44.0.0/24 via 172.25.132.35 dev tunl0 proto static onlink
10.44.1.0/24 via 172.25.132.55 dev tunl0 proto static onlink
10.44.2.0/24 dev cni0 proto kernel scope link src 10.44.2.1
10.44.2.0/24 dev kube-bridge proto kernel scope link src 10.44.2.1
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.25.132.0/23 dev eth0 proto kernel scope link src 172.25.132.230 metric 100
sudo ip a
28: kilo0: <POINTOPOINT,NOARP> mtu 1420 qdisc noqueue state DOWN group default qlen 1000
link/none
inet 10.4.0.1/16 brd 10.4.255.255 scope global kilo0
valid_lft forever preferred_lft forever
LOGS:
kubectl logs -n kube-system kilo-rm7xl
{"caller":"mesh.go:141","component":"kilo","level":"debug","msg":"using 172.25.132.230/23 as the private IP address","ts":"2021-07-22T09:42:22.29522959Z"}
{"caller":"mesh.go:146","component":"kilo","level":"debug","msg":"using 172.25.132.230/23 as the public IP address","ts":"2021-07-22T09:42:22.295347894Z"}
{"caller":"main.go:223","msg":"Starting Kilo network mesh '6309529a3ff0fd98a78ef2f352d5996387ef0293'.","ts":"2021-07-22T09:42:22.299384255Z"}
{"caller":"cni.go:60","component":"kilo","err":"failed to read IPAM config from CNI config list file: no IP ranges specified","level":"warn","msg":"failed to get CIDR from CNI file; overwriting it","ts":"2021-07-22T09:42:22.400952104Z"}
{"caller":"cni.go:68","component":"kilo","level":"info","msg":"CIDR in CNI file is empty","ts":"2021-07-22T09:42:22.401003506Z"}
{"CIDR":"10.44.2.0/24","caller":"cni.go:73","component":"kilo","level":"info","msg":"setting CIDR in CNI file","ts":"2021-07-22T09:42:22.401019107Z"}
{"caller":"mesh.go:268","component":"kilo","event":"add","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:42:22.509533933Z"}
{"caller":"mesh.go:279","component":"kilo","event":"add","in-mesh":false,"level":"debug","msg":"received non ready node","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"foundation-musanin-node-1","PersistentKeepalive":0,"Subnet":{"IP":"10.44.1.0","Mask":"////AA=="},"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-07-22T09:42:22.509630436Z"}
{"caller":"mesh.go:297","component":"kilo","event":"add","level":"info","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"foundation-musanin-node-1","PersistentKeepalive":0,"Subnet":{"IP":"10.44.1.0","Mask":"////AA=="},"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-07-22T09:42:22.509772342Z"}
{"caller":"mesh.go:268","component":"kilo","event":"add","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:42:22.509812444Z"}
{"caller":"mesh.go:270","component":"kilo","event":"add","level":"debug","msg":"processing local node","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"foundation-musanin-node-2","PersistentKeepalive":0,"Subnet":{"IP":"10.44.2.0","Mask":"////AA=="},"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-07-22T09:42:22.509824844Z"}
{"caller":"mesh.go:387","component":"kilo","level":"debug","msg":"local node differs from backend","ts":"2021-07-22T09:42:22.509857446Z"}
{"caller":"mesh.go:393","component":"kilo","level":"debug","msg":"successfully reconciled local node against backend","ts":"2021-07-22T09:42:22.520490269Z"}
{"DiscoveredEndpoints":{},"caller":"mesh.go:803","component":"kilo","level":"debug","msg":"Discovered WireGuard NAT Endpoints","ts":"2021-07-22T09:42:22.521797321Z"}
{"caller":"mesh.go:536","component":"kilo","level":"info","msg":"WireGuard configurations are different","ts":"2021-07-22T09:42:22.576117087Z"}
{"caller":"mesh.go:268","component":"kilo","event":"add","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:42:22.583246571Z"}
{"caller":"mesh.go:279","component":"kilo","event":"add","in-mesh":false,"level":"debug","msg":"received non ready node","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"foundation-musanin-master","PersistentKeepalive":0,"Subnet":{"IP":"10.44.0.0","Mask":"////AA=="},"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-07-22T09:42:22.583339575Z"}
{"caller":"mesh.go:297","component":"kilo","event":"add","level":"info","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"foundation-musanin-master","PersistentKeepalive":0,"Subnet":{"IP":"10.44.0.0","Mask":"////AA=="},"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-07-22T09:42:22.583374476Z"}
{"DiscoveredEndpoints":{},"caller":"mesh.go:803","component":"kilo","level":"debug","msg":"Discovered WireGuard NAT Endpoints","ts":"2021-07-22T09:42:22.584253011Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:42:22.585213549Z"}
{"caller":"mesh.go:270","component":"kilo","event":"update","level":"debug","msg":"processing local node","node":{"Endpoint":{"DNS":"","IP":"172.25.132.230","Port":51820},"Key":"SUVyaisrbGY4MGprV09FRVZzSDk3RzZ0VGJOR1ZpQ1oxMnMyR2VkbDVrZz0=","NoInternalIP":false,"InternalIP":{"IP":"172.25.132.230","Mask":"///+AA=="},"LastSeen":1626946942,"Leader":false,"Location":"","Name":"foundation-musanin-node-2","PersistentKeepalive":0,"Subnet":{"IP":"10.44.2.0","Mask":"////AA=="},"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":"location"},"ts":"2021-07-22T09:42:22.585255751Z"}
{"caller":"mesh.go:387","component":"kilo","level":"debug","msg":"local node differs from backend","ts":"2021-07-22T09:42:22.585350755Z"}
{"caller":"mesh.go:393","component":"kilo","level":"debug","msg":"successfully reconciled local node against backend","ts":"2021-07-22T09:42:22.596500099Z"}
{"DiscoveredEndpoints":{},"caller":"mesh.go:803","component":"kilo","level":"debug","msg":"Discovered WireGuard NAT Endpoints","ts":"2021-07-22T09:42:22.598070762Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:42:22.5990341Z"}
{"caller":"mesh.go:297","component":"kilo","event":"update","level":"info","node":{"Endpoint":{"DNS":"","IP":"172.25.132.55","Port":51820},"Key":"eUE2TGRDdUpUN3krcFJOdmxoZHM4R2VlRUdvVDFRL1BVaEYrK0daOGdCMD0=","NoInternalIP":false,"InternalIP":{"IP":"172.25.132.55","Mask":"///+AA=="},"LastSeen":1626946942,"Leader":false,"Location":"","Name":"foundation-musanin-node-1","PersistentKeepalive":0,"Subnet":{"IP":"10.44.1.0","Mask":"////AA=="},"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":"location"},"ts":"2021-07-22T09:42:22.599087803Z"}
{"DiscoveredEndpoints":{},"caller":"mesh.go:803","component":"kilo","level":"debug","msg":"Discovered WireGuard NAT Endpoints","ts":"2021-07-22T09:42:22.60003574Z"}
{"caller":"mesh.go:550","component":"kilo","level":"debug","msg":"local node is not the leader","ts":"2021-07-22T09:42:22.691546388Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:42:22.70363607Z"}
{"caller":"mesh.go:270","component":"kilo","event":"update","level":"debug","msg":"processing local node","node":{"Endpoint":{"DNS":"","IP":"172.25.132.230","Port":51820},"Key":"SUVyaisrbGY4MGprV09FRVZzSDk3RzZ0VGJOR1ZpQ1oxMnMyR2VkbDVrZz0=","NoInternalIP":false,"InternalIP":{"IP":"172.25.132.230","Mask":"///+AA=="},"LastSeen":1626946942,"Leader":false,"Location":"","Name":"foundation-musanin-node-2","PersistentKeepalive":0,"Subnet":{"IP":"10.44.2.0","Mask":"////AA=="},"WireGuardIP":{"IP":"10.4.0.1","Mask":"//8AAA=="},"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":"location"},"ts":"2021-07-22T09:42:22.703711473Z"}
{"caller":"mesh.go:387","component":"kilo","level":"debug","msg":"local node differs from backend","ts":"2021-07-22T09:42:22.703762875Z"}
{"caller":"mesh.go:393","component":"kilo","level":"debug","msg":"successfully reconciled local node against backend","ts":"2021-07-22T09:42:22.718906679Z"}
{"DiscoveredEndpoints":{},"caller":"mesh.go:803","component":"kilo","level":"debug","msg":"Discovered WireGuard NAT Endpoints","ts":"2021-07-22T09:42:22.721843296Z"}
kubectl logs -n kube-system kilo-tb5v9
{"caller":"mesh.go:356","component":"kilo","level":"debug","msg":"successfully checked in local node in backend","ts":"2021-07-22T09:56:22.875063564Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:56:22.87513167Z"}
{"caller":"mesh.go:270","component":"kilo","event":"update","level":"debug","msg":"processing local node","node":{"Endpoint":{"DNS":"","IP":"172.25.132.35","Port":51820},"Key":"Yzd5eWlTYUE5bnZMVkZ6NjBSa3I0Mit4ZHZDNEJWUGFHREtKKzV2NVFUVT0=","NoInternalIP":false,"InternalIP":{"IP":"172.25.132.35","Mask":"///+AA=="},"LastSeen":1626947782,"Leader":false,"Location":"","Name":"foundation-musanin-master","PersistentKeepalive":0,"Subnet":{"IP":"10.44.0.0","Mask":"////AA=="},"WireGuardIP":{"IP":"10.4.0.1","Mask":"//8AAA=="},"DiscoveredEndpoints":{},"AllowedLocationIPs":null,"Granularity":"location"},"ts":"2021-07-22T09:56:22.875154972Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:56:23.030299863Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:56:23.083005071Z"}
{"DiscoveredEndpoints":{},"caller":"mesh.go:803","component":"kilo","level":"debug","msg":"Discovered WireGuard NAT Endpoints","ts":"2021-07-22T09:56:52.682453593Z"}
{"caller":"mesh.go:356","component":"kilo","level":"debug","msg":"successfully checked in local node in backend","ts":"2021-07-22T09:56:52.883267527Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:56:52.884284105Z"}
{"caller":"mesh.go:270","component":"kilo","event":"update","level":"debug","msg":"processing local node","node":{"Endpoint":{"DNS":"","IP":"172.25.132.35","Port":51820},"Key":"Yzd5eWlTYUE5bnZMVkZ6NjBSa3I0Mit4ZHZDNEJWUGFHREtKKzV2NVFUVT0=","NoInternalIP":false,"InternalIP":{"IP":"172.25.132.35","Mask":"///+AA=="},"LastSeen":1626947812,"Leader":false,"Location":"","Name":"foundation-musanin-master","PersistentKeepalive":0,"Subnet":{"IP":"10.44.0.0","Mask":"////AA=="},"WireGuardIP":{"IP":"10.4.0.1","Mask":"//8AAA=="},"DiscoveredEndpoints":{},"AllowedLocationIPs":null,"Granularity":"location"},"ts":"2021-07-22T09:56:52.884331009Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:56:53.049789826Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:56:53.099891577Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:57:07.885698782Z"}
{"caller":"mesh.go:270","component":"kilo","event":"update","level":"debug","msg":"processing local node","node":{"Endpoint":{"DNS":"","IP":"172.25.132.35","Port":51820},"Key":"Yzd5eWlTYUE5bnZMVkZ6NjBSa3I0Mit4ZHZDNEJWUGFHREtKKzV2NVFUVT0=","NoInternalIP":false,"InternalIP":{"IP":"172.25.132.35","Mask":"///+AA=="},"LastSeen":1626947812,"Leader":false,"Location":"","Name":"foundation-musanin-master","PersistentKeepalive":0,"Subnet":{"IP":"10.44.0.0","Mask":"////AA=="},"WireGuardIP":{"IP":"10.4.0.1","Mask":"//8AAA=="},"DiscoveredEndpoints":{},"AllowedLocationIPs":null,"Granularity":"location"},"ts":"2021-07-22T09:57:07.885764487Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:57:22.398091373Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:57:22.398200282Z"}
{"caller":"mesh.go:270","component":"kilo","event":"update","level":"debug","msg":"processing local node","node":{"Endpoint":{"DNS":"","IP":"172.25.132.35","Port":51820},"Key":"Yzd5eWlTYUE5bnZMVkZ6NjBSa3I0Mit4ZHZDNEJWUGFHREtKKzV2NVFUVT0=","NoInternalIP":false,"InternalIP":{"IP":"172.25.132.35","Mask":"///+AA=="},"LastSeen":1626947812,"Leader":false,"Location":"","Name":"foundation-musanin-master","PersistentKeepalive":0,"Subnet":{"IP":"10.44.0.0","Mask":"////AA=="},"WireGuardIP":{"IP":"10.4.0.1","Mask":"//8AAA=="},"DiscoveredEndpoints":{},"AllowedLocationIPs":null,"Granularity":"location"},"ts":"2021-07-22T09:57:22.398216883Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:57:22.398270087Z"}
{"DiscoveredEndpoints":{},"caller":"mesh.go:803","component":"kilo","level":"debug","msg":"Discovered WireGuard NAT Endpoints","ts":"2021-07-22T09:57:22.68519494Z"}
{"caller":"mesh.go:356","component":"kilo","level":"debug","msg":"successfully checked in local node in backend","ts":"2021-07-22T09:57:22.889197319Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:57:22.889942876Z"}
{"caller":"mesh.go:270","component":"kilo","event":"update","level":"debug","msg":"processing local node","node":{"Endpoint":{"DNS":"","IP":"172.25.132.35","Port":51820},"Key":"Yzd5eWlTYUE5bnZMVkZ6NjBSa3I0Mit4ZHZDNEJWUGFHREtKKzV2NVFUVT0=","NoInternalIP":false,"InternalIP":{"IP":"172.25.132.35","Mask":"///+AA=="},"LastSeen":1626947842,"Leader":false,"Location":"","Name":"foundation-musanin-master","PersistentKeepalive":0,"Subnet":{"IP":"10.44.0.0","Mask":"////AA=="},"WireGuardIP":{"IP":"10.4.0.1","Mask":"//8AAA=="},"DiscoveredEndpoints":{},"AllowedLocationIPs":null,"Granularity":"location"},"ts":"2021-07-22T09:57:22.88999168Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:57:23.070523755Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:57:23.126743776Z"}
[k8s@foundation-musanin-master rke]$ clear
[k8s@foundation-musanin-master rke]$ kubectl logs -n kube-system kilo-tb5v9
{"caller":"mesh.go:141","component":"kilo","level":"debug","msg":"using 172.25.132.35/23 as the private IP address","ts":"2021-07-22T09:42:22.373165056Z"}
{"caller":"mesh.go:146","component":"kilo","level":"debug","msg":"using 172.25.132.35/23 as the public IP address","ts":"2021-07-22T09:42:22.373289366Z"}
{"caller":"main.go:223","msg":"Starting Kilo network mesh '6309529a3ff0fd98a78ef2f352d5996387ef0293'.","ts":"2021-07-22T09:42:22.377477892Z"}
{"caller":"cni.go:60","component":"kilo","err":"failed to read IPAM config from CNI config list file: no IP ranges specified","level":"warn","msg":"failed to get CIDR from CNI file; overwriting it","ts":"2021-07-22T09:42:22.478660578Z"}
{"caller":"cni.go:68","component":"kilo","level":"info","msg":"CIDR in CNI file is empty","ts":"2021-07-22T09:42:22.478718683Z"}
{"CIDR":"10.44.0.0/24","caller":"cni.go:73","component":"kilo","level":"info","msg":"setting CIDR in CNI file","ts":"2021-07-22T09:42:22.478736384Z"}
{"caller":"mesh.go:268","component":"kilo","event":"add","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:42:22.586319169Z"}
{"caller":"mesh.go:270","component":"kilo","event":"add","level":"debug","msg":"processing local node","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"foundation-musanin-master","PersistentKeepalive":0,"Subnet":{"IP":"10.44.0.0","Mask":"////AA=="},"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-07-22T09:42:22.586390874Z"}
{"caller":"mesh.go:387","component":"kilo","level":"debug","msg":"local node differs from backend","ts":"2021-07-22T09:42:22.586520985Z"}
{"caller":"mesh.go:393","component":"kilo","level":"debug","msg":"successfully reconciled local node against backend","ts":"2021-07-22T09:42:22.59864613Z"}
{"DiscoveredEndpoints":{},"caller":"mesh.go:803","component":"kilo","level":"debug","msg":"Discovered WireGuard NAT Endpoints","ts":"2021-07-22T09:42:22.60031746Z"}
{"caller":"mesh.go:536","component":"kilo","level":"info","msg":"WireGuard configurations are different","ts":"2021-07-22T09:42:22.653762925Z"}
{"caller":"mesh.go:268","component":"kilo","event":"add","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:42:22.656816763Z"}
{"caller":"mesh.go:279","component":"kilo","event":"add","in-mesh":false,"level":"debug","msg":"received non ready node","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"foundation-musanin-node-1","PersistentKeepalive":0,"Subnet":{"IP":"10.44.1.0","Mask":"////AA=="},"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-07-22T09:42:22.656874368Z"}
{"caller":"mesh.go:297","component":"kilo","event":"add","level":"info","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"foundation-musanin-node-1","PersistentKeepalive":0,"Subnet":{"IP":"10.44.1.0","Mask":"////AA=="},"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-07-22T09:42:22.656916371Z"}
{"DiscoveredEndpoints":{},"caller":"mesh.go:803","component":"kilo","level":"debug","msg":"Discovered WireGuard NAT Endpoints","ts":"2021-07-22T09:42:22.65869681Z"}
{"caller":"mesh.go:268","component":"kilo","event":"add","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:42:22.659554577Z"}
{"caller":"mesh.go:279","component":"kilo","event":"add","in-mesh":false,"level":"debug","msg":"received non ready node","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"foundation-musanin-node-2","PersistentKeepalive":0,"Subnet":{"IP":"10.44.2.0","Mask":"////AA=="},"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-07-22T09:42:22.65959418Z"}
{"caller":"mesh.go:297","component":"kilo","event":"add","level":"info","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"foundation-musanin-node-2","PersistentKeepalive":0,"Subnet":{"IP":"10.44.2.0","Mask":"////AA=="},"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-07-22T09:42:22.659635883Z"}
{"DiscoveredEndpoints":{},"caller":"mesh.go:803","component":"kilo","level":"debug","msg":"Discovered WireGuard NAT Endpoints","ts":"2021-07-22T09:42:22.661288012Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:42:22.662964542Z"}
{"caller":"mesh.go:297","component":"kilo","event":"update","level":"info","node":{"Endpoint":{"DNS":"","IP":"172.25.132.230","Port":51820},"Key":"SUVyaisrbGY4MGprV09FRVZzSDk3RzZ0VGJOR1ZpQ1oxMnMyR2VkbDVrZz0=","NoInternalIP":false,"InternalIP":{"IP":"172.25.132.230","Mask":"///+AA=="},"LastSeen":1626946942,"Leader":false,"Location":"","Name":"foundation-musanin-node-2","PersistentKeepalive":0,"Subnet":{"IP":"10.44.2.0","Mask":"////AA=="},"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":"location"},"ts":"2021-07-22T09:42:22.663004746Z"}
{"DiscoveredEndpoints":{},"caller":"mesh.go:803","component":"kilo","level":"debug","msg":"Discovered WireGuard NAT Endpoints","ts":"2021-07-22T09:42:22.66383051Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:42:22.749645698Z"}
{"caller":"mesh.go:297","component":"kilo","event":"update","level":"info","node":{"Endpoint":{"DNS":"","IP":"172.25.132.55","Port":51820},"Key":"eUE2TGRDdUpUN3krcFJOdmxoZHM4R2VlRUdvVDFRL1BVaEYrK0daOGdCMD0=","NoInternalIP":false,"InternalIP":{"IP":"172.25.132.55","Mask":"///+AA=="},"LastSeen":1626946942,"Leader":false,"Location":"","Name":"foundation-musanin-node-1","PersistentKeepalive":0,"Subnet":{"IP":"10.44.1.0","Mask":"////AA=="},"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":"location"},"ts":"2021-07-22T09:42:22.749812511Z"}
{"DiscoveredEndpoints":{},"caller":"mesh.go:803","component":"kilo","level":"debug","msg":"Discovered WireGuard NAT Endpoints","ts":"2021-07-22T09:42:22.750900396Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:42:22.872869802Z"}
{"caller":"mesh.go:297","component":"kilo","event":"update","level":"info","node":{"Endpoint":{"DNS":"","IP":"172.25.132.230","Port":51820},"Key":"SUVyaisrbGY4MGprV09FRVZzSDk3RzZ0VGJOR1ZpQ1oxMnMyR2VkbDVrZz0=","NoInternalIP":false,"InternalIP":{"IP":"172.25.132.230","Mask":"///+AA=="},"LastSeen":1626946942,"Leader":false,"Location":"","Name":"foundation-musanin-node-2","PersistentKeepalive":0,"Subnet":{"IP":"10.44.2.0","Mask":"////AA=="},"WireGuardIP":{"IP":"10.4.0.1","Mask":"//8AAA=="},"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":"location"},"ts":"2021-07-22T09:42:22.872944408Z"}
{"DiscoveredEndpoints":{},"caller":"mesh.go:803","component":"kilo","level":"debug","msg":"Discovered WireGuard NAT Endpoints","ts":"2021-07-22T09:42:22.874607538Z"}
kubectl logs -n kube-system kilo-tvgpl
{"caller":"mesh.go:550","component":"kilo","level":"debug","msg":"local node is not the leader","ts":"2021-07-22T09:57:23.114232984Z"}
{"caller":"mesh.go:561","component":"kilo","error":"failed to delete rule: no such file or directory","level":"error","ts":"2021-07-22T09:57:23.114893035Z"}
{"caller":"mesh.go:356","component":"kilo","level":"debug","msg":"successfully checked in local node in backend","ts":"2021-07-22T09:57:23.126617742Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:57:23.127445406Z"}
{"caller":"mesh.go:270","component":"kilo","event":"update","level":"debug","msg":"processing local node","node":{"Endpoint":{"DNS":"","IP":"172.25.132.55","Port":51820},"Key":"eUE2TGRDdUpUN3krcFJOdmxoZHM4R2VlRUdvVDFRL1BVaEYrK0daOGdCMD0=","NoInternalIP":false,"InternalIP":{"IP":"172.25.132.55","Mask":"///+AA=="},"LastSeen":1626947843,"Leader":false,"Location":"","Name":"foundation-musanin-node-1","PersistentKeepalive":0,"Subnet":{"IP":"10.44.1.0","Mask":"////AA=="},"WireGuardIP":null,"DiscoveredEndpoints":{},"AllowedLocationIPs":null,"Granularity":"location"},"ts":"2021-07-22T09:57:23.127482909Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:57:52.896760817Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:57:53.087422463Z"}
{"DiscoveredEndpoints":{},"caller":"mesh.go:803","component":"kilo","level":"debug","msg":"Discovered WireGuard NAT Endpoints","ts":"2021-07-22T09:57:53.117206066Z"}
{"caller":"mesh.go:550","component":"kilo","level":"debug","msg":"local node is not the leader","ts":"2021-07-22T09:57:53.130811719Z"}
{"caller":"mesh.go:561","component":"kilo","error":"failed to delete rule: no such file or directory","level":"error","ts":"2021-07-22T09:57:53.131117342Z"}
{"caller":"mesh.go:356","component":"kilo","level":"debug","msg":"successfully checked in local node in backend","ts":"2021-07-22T09:57:53.143393992Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:57:53.143484499Z"}
{"caller":"mesh.go:270","component":"kilo","event":"update","level":"debug","msg":"processing local node","node":{"Endpoint":{"DNS":"","IP":"172.25.132.55","Port":51820},"Key":"eUE2TGRDdUpUN3krcFJOdmxoZHM4R2VlRUdvVDFRL1BVaEYrK0daOGdCMD0=","NoInternalIP":false,"InternalIP":{"IP":"172.25.132.55","Mask":"///+AA=="},"LastSeen":1626947873,"Leader":false,"Location":"","Name":"foundation-musanin-node-1","PersistentKeepalive":0,"Subnet":{"IP":"10.44.1.0","Mask":"////AA=="},"WireGuardIP":null,"DiscoveredEndpoints":{},"AllowedLocationIPs":null,"Granularity":"location"},"ts":"2021-07-22T09:57:53.143518001Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:58:08.370304068Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:58:22.907335487Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:58:23.101513105Z"}
{"DiscoveredEndpoints":{},"caller":"mesh.go:803","component":"kilo","level":"debug","msg":"Discovered WireGuard NAT Endpoints","ts":"2021-07-22T09:58:23.132699317Z"}
{"caller":"mesh.go:550","component":"kilo","level":"debug","msg":"local node is not the leader","ts":"2021-07-22T09:58:23.157593343Z"}
{"caller":"mesh.go:561","component":"kilo","error":"failed to delete rule: no such file or directory","level":"error","ts":"2021-07-22T09:58:23.158234692Z"}
{"caller":"mesh.go:356","component":"kilo","level":"debug","msg":"successfully checked in local node in backend","ts":"2021-07-22T09:58:23.164685391Z"}
{"caller":"mesh.go:268","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-07-22T09:58:23.166577337Z"}
{"caller":"mesh.go:270","component":"kilo","event":"update","level":"debug","msg":"processing local node","node":{"Endpoint":{"DNS":"","IP":"172.25.132.55","Port":51820},"Key":"eUE2TGRDdUpUN3krcFJOdmxoZHM4R2VlRUdvVDFRL1BVaEYrK0daOGdCMD0=","NoInternalIP":false,"InternalIP":{"IP":"172.25.132.55","Mask":"///+AA=="},"LastSeen":1626947903,"Leader":false,"Location":"","Name":"foundation-musanin-node-1","PersistentKeepalive":0,"Subnet":{"IP":"10.44.1.0","Mask":"////AA=="},"WireGuardIP":null,"DiscoveredEndpoints":{},"AllowedLocationIPs":null,"Granularity":"location"},"ts":"2021-07-22T09:58:23.166620741Z"}
Hi @squat, I strongly believe in your project and hope you could help me with the final issue...
I have successfully installed kilo on my RKE cluster as a CNI:
master-node
have access to kube-apiserver endpointbut I have caught another issue: PODs in the
non-master
nodes CANNOT access to kube-apiserver (kubernetes.default -> 10.45.0.1:443 -> 172.25.132.35:6443)It is critical for k8s operators like Istio, Prometheus, Infinispan, etc. and I got stuck with that...
I guess something wrong with network routing (KILO-NAT, KILO-IPIP iptables). Please check my k8s configuration, test PODs, and iptables of
master-node
andnode-1
, probably you might find a reason:kubectl get nodes -o wide
kubectl get pods --all-namespaces -o wide
kubectl get service --all-namespaces -o wide
kubectl get endpoints --all-namespaces -o wide
echoserver POD is accessible from all nodes
but
kube-apiserver
API is accessible only from MASTER POD !!!below network settings of master-node (foundation-musanin-master)
sudo iptables-save > ~/temp/20210720_iptables_master
sudo ifconfig
sudo ip a
sudo ip r
sudo wg
below network settings of node-1 (foundation-musanin-node-1)
sudo iptables-save > ~/temp/20210720_iptables_node1
sudo ifconfig
sudo ip a
sudo ip r
sudo wg
kgctl graph
I got stuck with iptables routing and unclear how to set up access from
node-1 (foundation-musanin-node-1)
tokube-apiserver (kubernetes.default -> 10.45.0.1:443 -> 172.25.132.35:6443)
Hope for your help.
Thanks in advance, Vladimir.