ElasticSearch on Kubernetes, one master nod not discovered among three

I have deployed an elasticsearch in Kubernetes with 3 masters and 3 data nodes. All the pods are running.

NAME                        READY   STATUS    RESTARTS   AGE
elasticsearch-master-v1-0   1/1     Running   0          29m
elasticsearch-master-v1-1   1/1     Running   0          29m
elasticsearch-master-v1-2   1/1     Running   0          29m

Note: Now the replicaset for data-node deployment is set to 0.

Elasticsearch Version: 6.4.0

But when I had checked the the node status through elasticsearch API, only two nodes are showing up.

API CALL: /_cat/nodes?v

ip              heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.188.158.230            2          38  27    0.35    0.21     0.15 m         *      elasticsearch-master-v1-1
192.188.119.232            2          38  19    0.55    0.26     0.17 m         -      elasticsearch-master-v1-0

It should show all the three master nodes above

Why the third node is not showing up here? The following are the relevant configuration in elasticsearch.yml file.

cluster:
  name: ${CLUSTER_NAME}

node:
  master: true
  data: false
  name: ${NODE_NAME}
  ingest: false

processors: ${PROCESSORS}

network.host: 0.0.0.0

path:
  data: ${DATA_PATH}
  logs: ${LOG_PATH}

bootstrap:
  memory_lock: ${MEMORY_LOCK}

http:
  compression: true
  cors:
    enabled: true
    allow-origin: "*"

discovery:
  zen:
    ping.unicast.hosts: elasticsearch-discovery-master
    minimum_master_nodes: 2

xpack.ml.enabled: false
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12

I noticed that when in the above configuration xpack.security.transport.ssl.enabled: false on the same API call it showed all the three master nodes

K8s Service yaml for inter node communication.

apiVersion: v1
kind: Service
metadata:
  labels:
    app: elasticsearch-master
  name: elasticsearch-discovery-master
spec:
  ports:
  - name: transport
    port: 9300
    protocol: TCP
    targetPort: 9300
  selector:
    app: elasticsearch-master
  type: ClusterIP

This is what the logs says from the third master node which is not discovered,

[2020-08-12T12:38:41,342][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-master-v1-2] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-master-v1-2}{F8L_VgGuRKWxxjKcQzhfzg}{ZPk-SAFESX2UFsEcH6Q7Hw}{192.188.193.101}{192.188.193.101:9300}{xpack.installed=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2020-08-12T12:38:50,616][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-master-v1-2] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-master-v1-2}{F8L_VgGuRKWxxjKcQzhfzg}{ZPk-SAFESX2UFsEcH6Q7Hw}{192.188.193.101}{192.188.193.101:9300}{xpack.installed=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2020-08-12T12:39:15,424][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-master-v1-2] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-master-v1-2}{F8L_VgGuRKWxxjKcQzhfzg}{ZPk-SAFESX2UFsEcH6Q7Hw}{192.188.193.101}{192.188.193.101:9300}{xpack.installed=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2020-08-12T12:39:18,425][WARN ][o.e.d.z.ZenDiscovery     ] [elasticsearch-master-v1-2] not enough master nodes discovered during pinging (found [[Candidate{node={elasticsearch-master-v1-2}{F8L_VgGuRKWxxjKcQzhfzg}{ZPk-SAFESX2UFsEcH6Q7Hw}{192.188.193.101}{192.188.193.101:9300}{xpack.installed=true}, clusterStateVersion=-1}]], but needed [2]), pinging again

elastic / elasticsearch

ElasticSearch on Kubernetes, one master nod not discovered among three #61330