search

projectcalico / calico

Cloud native networking and network security
https://docs.tigera.io/calico/latest/about/
Apache License 2.0
5.89k stars 1.31k forks source link

CoreDNS logs i/o timeout in ipv6 only mode #3673

Closed lynic closed 3 years ago

lynic commented 4 years ago

I setup a ipv6 only k8s with calico, and can create pods and ping each other. But when I look at coreDNS logs, it reports some 443 i/o timeout errors.

Expected Behavior

coreDNS shouldn't report 443 i/o timeout errors

Current Behavior

pods are running

vagrant@ubuntu:~$ kubectl get pods -A -owide
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE   IP                         NODE     NOMINATED NODE   READINESS GATES
default       pod1                                       1/1     Running   0          21s   fd20::f3c4                 ubuntu   <none>           <none>
default       pod2                                       1/1     Running   0          21s   fd20::f3c3                 ubuntu   <none>           <none>
kube-system   calico-kube-controllers-76d4774d89-2fw2f   1/1     Running   1          26m   fd20::f3c0                 ubuntu   <none>           <none>
kube-system   calico-node-6s76r                          1/1     Running   0          26m   2020::250:56ff:feb9:4cf2   ubuntu   <none>           <none>
kube-system   coredns-66bff467f8-fcndb                   1/1     Running   0          26m   fd20::f3c1                 ubuntu   <none>           <none>
kube-system   coredns-66bff467f8-hg57w                   1/1     Running   0          26m   fd20::f3c2                 ubuntu   <none>           <none>
kube-system   etcd-ubuntu                                1/1     Running   0          27m   2020::250:56ff:feb9:4cf2   ubuntu   <none>           <none>
kube-system   kube-apiserver-ubuntu                      1/1     Running   0          27m   2020::250:56ff:feb9:4cf2   ubuntu   <none>           <none>
kube-system   kube-controller-manager-ubuntu             1/1     Running   0          27m   2020::250:56ff:feb9:4cf2   ubuntu   <none>           <none>
kube-system   kube-proxy-p5xsq                           1/1     Running   0          26m   2020::250:56ff:feb9:4cf2   ubuntu   <none>           <none>
kube-system   kube-scheduler-ubuntu                      1/1     Running   0          27m   2020::250:56ff:feb9:4cf2   ubuntu   <none>           <none>

vagrant@ubuntu:~$ kubectl exec -it pod1 ping6 fd20::f3c3
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
PING fd20::f3c3(fd20::f3c3) 56 data bytes
64 bytes from fd20::f3c3: icmp_seq=1 ttl=63 time=0.258 ms
64 bytes from fd20::f3c3: icmp_seq=2 ttl=63 time=0.085 ms

vagrant@ubuntu:~$ kubectl get svc -A 
NAMESPACE     NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       kubernetes   ClusterIP   fd03::1      <none>        443/TCP                  27m
kube-system   kube-dns     ClusterIP   fd03::a      <none>        53/UDP,53/TCP,9153/TCP   27m

ip6tables rules

vagrant@ubuntu:~$ sudo ip6tables -t nat -L -n|grep 443
DNAT       tcp      ::/0                 ::/0                 /* default/kubernetes:https */ tcp to:[2020::250:56ff:feb9:4cf2]:6443
KUBE-MARK-MASQ  tcp     !fd20::/112           fd03::1              /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-SVC-NPX46M4PTMTKRN6Y  tcp      ::/0                 fd03::1              /* default/kubernetes:https cluster IP */ tcp dpt:443

coredns logs

vagrant@ubuntu:~$ kubectl logs -f coredns-66bff467f8-fcndb -n kube-system
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.7
linux/amd64, go1.13.6, da7f65b
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
I0618 13:20:47.153421       1 trace.go:116] Trace[2019727887]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-06-18 13:20:17.152198639 +0000 UTC m=+0.039360227) (total time: 30.00108793s):
Trace[2019727887]: [30.00108793s] [30.00108793s] END
E0618 13:20:47.153472       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Service: Get https://[fd03::1]:443/api/v1/services?limit=500&resourceVersion=0: dial tcp [fd03::1]:443: i/o timeout
I0618 13:20:47.153510       1 trace.go:116] Trace[1427131847]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-06-18 13:20:17.15188227 +0000 UTC m=+0.039043907) (total time: 30.001565063s):
Trace[1427131847]: [30.001565063s] [30.001565063s] END
E0618 13:20:47.153520       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Namespace: Get https://[fd03::1]:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp [fd03::1]:443: i/o timeout
I0618 13:20:47.153838       1 trace.go:116] Trace[939984059]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-06-18 13:20:17.152054987 +0000 UTC m=+0.039216598) (total time: 30.001747881s):
Trace[939984059]: [30.001747881s] [30.001747881s] END
E0618 13:20:47.153851       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://[fd03::1]:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp [fd03::1]:443: i/o timeout

Possible Solution

Steps to Reproduce (for bugs)

See above logs

Context

Your Environment

caseydavenport commented 4 years ago

These looks like errors trying to access the Kubernetes API service:

E0618 13:20:47.153851 1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://[fd03::1]:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp [fd03::1]:443: i/o timeout

Have you confirmed that the service is working properly and that the host's IPv6 address is reachable? First thing to do is confirm that services are working properly, since it sounds like pod->pod is functioning as expected

mgleung commented 3 years ago

Closing because this is stale. Please reopen if more issues occur.