k8s部署elastic7.10.0报错不能识别pod的DNS

[githublwk087256]

在AzureVM中自建了k8s集群,一台master节点,三台worker节点,在k8s集群中部署了elastic服务,启动了三个pod,k8s的网络使用的是calico,目前k8s集群启动成功,elastic的pod也启动成功,但是报错不能识别其他两个pod的DNS {"type": "server", "timestamp": "2024-04-01T01:22:06,506Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "myk8se": "elastic-cluster-ctl-0", "message": "timed out after [5s] resolving host [elastic-cluster-master-0.es7-headless.elastic-worker.svc.clus {"type": "server", "timestamp": "2024-04-01T01:22:12,506Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "myk8se": "elastic-cluster-ctl-0", "message": "failed to resolve host [elastic-cluster-data-0.es7-headless.elastic-worker.svc.cluster.local]", "stacktrace": ["java.net.UnknownHostException: elastic-cluster-data-0.es7-headless.elastic-worker.svc.cluster.local", "at java.net.InetAddress$CachedAddresses.get(InetAddress.java:800) ~[?:?]", "at java.net.InetAddress.getAllByName0(InetAddress.java:1507) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1366) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1300) ~[?:?]", "at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:556) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:498) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:864) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:144) ~[elasticsearch-7.10.0.jar:7.10.0] "at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.10.0.jar "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", "at java.lang.Thread.run(Thread.java:832) [?:?]"] } {"type": "server", "timestamp": "2024-04-01T01:22:12,507Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "myk8se": "elastic-cluster-ctl-0", "message": "timed out after [5s] resolving host [elastic-cluster-master-0.es7-headless.elastic-worker.svc.clus {"type": "server", "timestamp": "2024-04-01T01:22:15,519Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "myk8se": "elastic-cluster-ctl-0", "message": "failed to resolve host [elastic-cluster-data-0.es7-headless.elastic-worker.svc.cluster.local]", "stacktrace": ["java.net.UnknownHostException: elastic-cluster-data-0.es7-headless.elastic-worker.svc.cluster.local", "at java.net.InetAddress$CachedAddresses.get(InetAddress.java:800) ~[?:?]", "at java.net.InetAddress.getAllByName0(InetAddress.java:1507) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1366) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1300) ~[?:?]", "at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:556) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:498) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:864) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:144) ~[elasticsearch-7.10.0.jar:7.10.0] "at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.10.0.jar "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", "at java.lang.Thread.run(Thread.java:832) [?:?]"] } {"type": "server", "timestamp": "2024-04-01T01:22:15,520Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "myk8se": "elastic-cluster-ctl-0", "message": "failed to resolve host [elastic-cluster-master-0.es7-headless.elastic-worker.svc.cluster.local]", "stacktrace": ["java.net.UnknownHostException: elastic-cluster-master-0.es7-headless.elastic-worker.svc.cluster.local", "at java.net.InetAddress$CachedAddresses.get(InetAddress.java:800) ~[?:?]", "at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:886) ~[?:?]", "at java.net.InetAddress.getAllByName0(InetAddress.java:1507) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1366) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1300) ~[?:?]", "at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:556) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:498) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:864) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:144) ~[elasticsearch-7.10.0.jar:7.10.0] "at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.10.0.jar "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", "at java.lang.Thread.run(Thread.java:832) [?:?]"] } {"type": "server", "timestamp": "2024-04-01T01:22:21,508Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "myk8se": "elastic-cluster-ctl-0", "message": "timed out after [5s] resolving host [elastic-cluster-data-0.es7-headless.elastic-worker.svc.cluste {"type": "server", "timestamp": "2024-04-01T01:22:21,508Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "myk8se": "elastic-cluster-ctl-0", "message": "failed to resolve host [elastic-cluster-master-0.es7-headless.elastic-worker.svc.cluster.local]", "stacktrace": ["java.net.UnknownHostException: elastic-cluster-master-0.es7-headless.elastic-worker.svc.cluster.local", "at java.net.InetAddress$CachedAddresses.get(InetAddress.java:800) ~[?:?]", "at java.net.InetAddress.getAllByName0(InetAddress.java:1507) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1366) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1300) ~[?:?]", "at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:556) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:498) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:864) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:144) ~[elasticsearch-7.10.0.jar:7.10.0] "at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.10.0.jar "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", "at java.lang.Thread.run(Thread.java:832) [?:?]"] } {"type": "server", "timestamp": "2024-04-01T01:22:27,509Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "myk8se": "elastic-cluster-ctl-0", "message": "timed out after [5s] resolving host [elastic-cluster-data-0.es7-headless.elastic-worker.svc.cluste {"type": "server", "timestamp": "2024-04-01T01:22:27,509Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "myk8se": "elastic-cluster-ctl-0", "message": "failed to resolve host [elastic-cluster-master-0.es7-headless.elastic-worker.svc.cluster.local]", "stacktrace": ["java.net.UnknownHostException: elastic-cluster-master-0.es7-headless.elastic-worker.svc.cluster.local", "at java.net.InetAddress$CachedAddresses.get(InetAddress.java:800) ~[?:?]", "at java.net.InetAddress.getAllByName0(InetAddress.java:1507) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1366) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1300) ~[?:?]", "at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:556) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:498) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:864) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:144) ~[elasticsearch-7.10.0.jar:7.10.0] "at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.10.0.jar "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", "at java.lang.Thread.run(Thread.java:832) [?:?]"] } {"type": "server", "timestamp": "2024-04-01T01:22:33,510Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "myk8se": "elastic-cluster-ctl-0", "message": "timed out after [5s] resolving host [elastic-cluster-data-0.es7-headless.elastic-worker.svc.cluste {"type": "server", "timestamp": "2024-04-01T01:22:33,510Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "myk8se": "elastic-cluster-ctl-0", "message": "failed to resolve host [elastic-cluster-master-0.es7-headless.elastic-worker.svc.cluster.local]", "stacktrace": ["java.net.UnknownHostException: elastic-cluster-master-0.es7-headless.elastic-worker.svc.cluster.local", "at java.net.InetAddress$CachedAddresses.get(InetAddress.java:800) ~[?:?]", "at java.net.InetAddress.getAllByName0(InetAddress.java:1507) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1366) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1300) ~[?:?]", "at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:556) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:498) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:864) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:144) ~[elasticsearch-7.10.0.jar:7.10.0] "at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.10.0.jar "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", "at java.lang.Thread.run(Thread.java:832) [?:?]"] } {"type": "server", "timestamp": "2024-04-01T01:22:39,511Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "myk8se": "elastic-cluster-ctl-0", "message": "failed to resolve host [elastic-cluster-data-0.es7-headless.elastic-worker.svc.cluster.local]", "stacktrace": ["java.net.UnknownHostException: elastic-cluster-data-0.es7-headless.elastic-worker.svc.cluster.local", "at java.net.InetAddress$CachedAddresses.get(InetAddress.java:800) ~[?:?]", "at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:886) ~[?:?]", "at java.net.InetAddress.getAllByName0(InetAddress.java:1507) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1366) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1300) ~[?:?]", "at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:556) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:498) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:864) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:144) ~[elasticsearch-7.10.0.jar:7.10.0] "at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.10.0.jar "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", "at java.lang.Thread.run(Thread.java:832) [?:?]"] } {"type": "server", "timestamp": "2024-04-01T01:22:39,511Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "myk8se": "elastic-cluster-ctl-0", "message": "timed out after [5s] resolving host [elastic-cluster-master-0.es7-headless.elastic-worker.svc.clus {"type": "server", "timestamp": "2024-04-01T01:22:45,512Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "myk8se": "elastic-cluster-ctl-0", "message": "failed to resolve host [elastic-cluster-data-0.es7-headless.elastic-worker.svc.cluster.local]", "stacktrace": ["java.net.UnknownHostException: elastic-cluster-data-0.es7-headless.elastic-worker.svc.cluster.local", "at java.net.InetAddress$CachedAddresses.get(InetAddress.java:800) ~[?:?]", "at java.net.InetAddress.getAllByName0(InetAddress.java:1507) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1366) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1300) ~[?:?]", "at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:556) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:498) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:864) ~[elasticsearch-7.10.0.jar:7.10.0]", "at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:144) ~[elasticsearch-7.10.0.jar:7.10.0] "at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.10.0.jar "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]", "at java.lang.Thread.run(Thread.java:832) [?:?]"] }

[githublwk087256]

kubectl -n elastic-worker describe StatefulSet elastic-cluster-ctl Name: elastic-cluster-ctl Namespace: elastic-worker CreationTimestamp: Wed, 27 Mar 2024 08:04:08 +0000 Selector: app=elastic-cluster,kubernetes.io/cluster-service=true Labels: addonmanager.kubernetes.io/mode=Reconcile app=elastic-ctl kubernetes.io/cluster-service=true srv=srv-elasticsearch Annotations: Replicas: 1 desired | 1 total Update Strategy: RollingUpdate Partition: 0 Pods Status: 1 Running / 0 Waiting / 0 Succeeded / 0 Failed Pod Template: Labels: app=elastic-cluster kubernetes.io/cluster-service=true Service Account: elastic-cluster-serviceaccount Init Containers: fix-permissions: Image: busybox Port: Host Port: Environment: Mounts: increase-vm-max-map: Image: busybox Port: Host Port: Command: sysctl -w vm.max_map_count=262144 Environment: Mounts: increase-fd-ulimit: Image: busybox Port: Host Port: Command: sh -c ulimit -n 65536 Environment: Mounts: create-elastic-certpath: Image: busybox Port: Host Port: Command: sh -c mkdir -p /usr/share/elasticsearch/config/certs/ && chown -R 1000:1000 /usr/share/elasticsearch/config/certs/ Environment: Mounts: Containers: elasticsearch-ctl: Image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0 Ports: 9200/TCP, 9300/TCP Host Ports: 0/TCP, 0/TCP Limits: cpu: 1 memory: 2Gi Environment: POD_NAME: (v1:metadata.name) node.roles: ingest network.host: 0.0.0.0 path.data: /usr/share/elasticsearch/data/es_ctl_data path.logs: /usr/share/elasticsearch/logs/es_ctl_logs/ Mounts: /etc/localtime from localtime (ro) /usr/share/elasticsearch/config/certs/ from certificates-volume (ro) /usr/share/elasticsearch/config/elasticsearch.yml from config-volume (rw,path="elasticsearch.yml") /usr/share/elasticsearch/data/es_ctl_node_data/ from elastic-data (rw) /usr/share/elasticsearch/logs/es_ctl_node_logs/ from elastic-logs (rw) Volumes: elastic-data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: elasticsearch-ctl-node-data-pvc ReadOnly: false elastic-logs: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: elasticsearch-ctl-node-logs-pvc ReadOnly: false localtime: Type: HostPath (bare host directory volume) Path: /etc/localtime HostPathType: config-volume: Type: ConfigMap (a volume populated by a ConfigMap) Name: elasticsearch-config Optional: false certificates-volume: Type: Secret (a volume populated by a Secret) SecretName: elasticsearch-certs Optional: false Volume Claims: Events:

[githublwk087256]

经过排查,目前配置了 bgp、ipvs,发现worker节点上的pod相互不能ping通,但是对应节点上的宿主机与pod能ping通,master节点与worker节点中的podIPping不通