search

etcd-io / etcd

Distributed reliable key-value store for the most critical data of a distributed system
https://etcd.io
Apache License 2.0
47.22k stars 9.7k forks source link

etcd pods restart stating etcddomain reject: does not match any of DNSNames #10572

Closed ghost closed 3 years ago

ghost commented 5 years ago

Team, I have 3 masters of which one was manually deleted, and it was brought up by auto scale group however, etcd started to throw below on the new master that joined cluster. Any hint?

etcdmain: the server is already initialized as member before, starting as etcd member... 2019-03-21 16:32:25.541529 I | embed: peerTLS: cert = /srv/kubernetes/etcd.pem, key = /srv/kubernetes/etcd-key.pem, ca = , trusted-ca = /srv/kubernetes/ca.crt, client-cert-auth = true 2019-03-21 16:32:25.542119 I | embed: listening for peers on https://0.0.0.0:2380 2019-03-21 16:32:25.542161 I | embed: listening for client requests on 0.0.0.0:4001 2019-03-21 16:32:25.554167 I | pkg/netutil: resolving etcd-b.internal.test.com:2380 to 11.4.0.26:2380 2019-03-21 16:32:25.564718 I | pkg/netutil: resolving etcd-b.internal.test.com:2380 to 11.4.0.26:2380 2019-03-21 16:32:25.649568 I | etcdmain: rejected connection from "11.4.0.4:40736" (error "tls: \"11.4.0.4\" does not match any of DNSNames [\"etcd-a.internal.test.com\" \"etcd-b.internal.test.com\" \"etcd-c.internal.test.com\" \"etcd-events-a.internal.test.com\" \"etcd-events-b.internal.test.com\" \"etcd-events-c.internal.test.com\" \"localhost\"] (lookup etcd-events-c.internal.test.com on 100.64.0.10:53: dial udp 100.64.0.10:53: operation was canceled)", ServerName "etcd-b.internal.test.com", IPAddresses ["127.0.0.1"], DNSNames ["etcd-a.internal.test.com" "etcd-b.internal.test.com" "etcd-c.internal.test.com" "etcd-events-a.internal.test.com" "etcd-events-b.internal.test.com" "etcd-events-c.internal.test.com" "localhost"]) 2019-03-21 16:32:25.683192 C | etcdmain: member 16acea4929706d3 has already been bootstrapped

etcd.pem shows below X509v3 Subject Alternative Name: DNS:etcd-a.internal.test.com, DNS:etcd-b.internal.test.com, DNS:etcd-c.internal.test.com, DNS:etcd-events-a.internal.test.com, DNS:etcd-events-b.internal.test.com, DNS:etcd-events-c.internal.test.com, DNS:localhost, IP Address:127.0.0.1

hexfusion commented 5 years ago

etcdmain: rejected connection from "11.4.0.4:40736" (error "tls: "11.4.0.4" does not match any of DNSNames ["etcd-a.internal.test.com" "etcd-b.internal.test.com" "etcd-c.internal.test.com" "etcd-events-a.internal.test.com" "etcd-events-b.internal.test.com" "etcd-events-c.internal.test.com" "localhost"] (lookup etcd-events-c.internal.test.com on 100.64.0.10:53:

what is the output of

dig +short etcd-a.internal.test.com etcd-b.internal.test.com etcd-c.internal.test.com etcd-events-a.internal.test.com etcd-events-b.internal.test.com etcd-events-c.internal.test.com

unless 11.4.0.4 matches one of these domains the error is literal

hexfusion commented 5 years ago

@ctxrag just touching base on this any updates per above?

jpbetz commented 4 years ago

xref https://github.com/kubernetes/kubernetes/issues/81508#issuecomment-590646553

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.