Default Elasticsearch Installation stuck on "readiness probe failed"

[melvin-suter]

Bug Report

What did you do?

I installed ECK without any issues. (Kibana for instance works fine.) I used this config to create an elasticsearch instance:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elastic
spec:
  version: 8.6.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: nfs-rwo

What did you expect to see?

A running and ready elasticsearch pod.

What did you see instead? Under which circumstances?

The pod starts and is shown as "running", but I can't seem to get it to work. The logs of the pod get filled with this entry:

{"timestamp": "2023-01-19T22:50:37+00:00", "message": "readiness probe failed", "curl_rc": "7"}

If I open a shell inside the pod I cant curl 172.0.0.1:9200 .

Environment

• ECK version:

  2.6

• Kubernetes information:

I'm running a RKE2 on premise cluster. v1.24.8+rke2r1

• Logs:

  Skipping security auto configuration because the configuration file [/usr/share/elasticsearch/config/elasticsearch.yml] is missing or is not a regular file
  {"@timestamp":"2023-01-19T22:49:58.854Z", "log.level": "INFO", "message":"version[8.6.0], pid[68], build[docker/f67ef2df40237445caa70e2fef79471cc608d70d/2023-01-04T09:35:21.782467981Z], OS[Linux/4.18.0-348.el8.0.2.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/19.0.1/19.0.1+10-21]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:49:58.858Z", "log.level": "INFO", "message":"JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:49:58.858Z", "log.level": "INFO", "message":"JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-9843315506962037583, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1024m, -Xmx1024m, -XX:MaxDirectMemorySize=536870912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.015Z", "log.level": "INFO", "message":"Package versions: jackson-annotations=2.13.2, jackson-core=2.13.2, jackson-databind=2.13.2.2, jackson-dataformat-xml=2.13.2, jackson-datatype-jsr310=2.13.2, azure-core=1.27.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"com.azure.core.implementation.jackson.JacksonVersion","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.970Z", "log.level": "INFO", "message":"loaded module [aggregations]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.970Z", "log.level": "INFO", "message":"loaded module [aggs-matrix-stats]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.970Z", "log.level": "INFO", "message":"loaded module [analysis-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.970Z", "log.level": "INFO", "message":"loaded module [apm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.970Z", "log.level": "INFO", "message":"loaded module [constant-keyword]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.970Z", "log.level": "INFO", "message":"loaded module [data-streams]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.970Z", "log.level": "INFO", "message":"loaded module [frozen-indices]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.971Z", "log.level": "INFO", "message":"loaded module [ingest-attachment]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.971Z", "log.level": "INFO", "message":"loaded module [ingest-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.971Z", "log.level": "INFO", "message":"loaded module [ingest-geoip]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.971Z", "log.level": "INFO", "message":"loaded module [ingest-user-agent]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.971Z", "log.level": "INFO", "message":"loaded module [kibana]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.971Z", "log.level": "INFO", "message":"loaded module [lang-expression]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.971Z", "log.level": "INFO", "message":"loaded module [lang-mustache]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.971Z", "log.level": "INFO", "message":"loaded module [lang-painless]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.971Z", "log.level": "INFO", "message":"loaded module [legacy-geo]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.971Z", "log.level": "INFO", "message":"loaded module [mapper-extras]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.972Z", "log.level": "INFO", "message":"loaded module [mapper-version]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.972Z", "log.level": "INFO", "message":"loaded module [old-lucene-versions]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.972Z", "log.level": "INFO", "message":"loaded module [parent-join]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.972Z", "log.level": "INFO", "message":"loaded module [percolator]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.972Z", "log.level": "INFO", "message":"loaded module [rank-eval]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.972Z", "log.level": "INFO", "message":"loaded module [reindex]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.972Z", "log.level": "INFO", "message":"loaded module [repositories-metering-api]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.972Z", "log.level": "INFO", "message":"loaded module [repository-azure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.972Z", "log.level": "INFO", "message":"loaded module [repository-encrypted]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.972Z", "log.level": "INFO", "message":"loaded module [repository-gcs]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.972Z", "log.level": "INFO", "message":"loaded module [repository-s3]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.973Z", "log.level": "INFO", "message":"loaded module [repository-url]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.973Z", "log.level": "INFO", "message":"loaded module [runtime-fields-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.973Z", "log.level": "INFO", "message":"loaded module [search-business-rules]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.973Z", "log.level": "INFO", "message":"loaded module [searchable-snapshots]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.973Z", "log.level": "INFO", "message":"loaded module [snapshot-based-recoveries]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.973Z", "log.level": "INFO", "message":"loaded module [snapshot-repo-test-kit]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.973Z", "log.level": "INFO", "message":"loaded module [spatial]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.973Z", "log.level": "INFO", "message":"loaded module [transform]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.973Z", "log.level": "INFO", "message":"loaded module [transport-netty4]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.974Z", "log.level": "INFO", "message":"loaded module [unsigned-long]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.974Z", "log.level": "INFO", "message":"loaded module [vector-tile]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.974Z", "log.level": "INFO", "message":"loaded module [wildcard]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.974Z", "log.level": "INFO", "message":"loaded module [x-pack-aggregate-metric]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.974Z", "log.level": "INFO", "message":"loaded module [x-pack-analytics]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.974Z", "log.level": "INFO", "message":"loaded module [x-pack-async]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.974Z", "log.level": "INFO", "message":"loaded module [x-pack-async-search]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.974Z", "log.level": "INFO", "message":"loaded module [x-pack-autoscaling]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.974Z", "log.level": "INFO", "message":"loaded module [x-pack-ccr]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.974Z", "log.level": "INFO", "message":"loaded module [x-pack-core]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.975Z", "log.level": "INFO", "message":"loaded module [x-pack-deprecation]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.975Z", "log.level": "INFO", "message":"loaded module [x-pack-enrich]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.975Z", "log.level": "INFO", "message":"loaded module [x-pack-eql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.975Z", "log.level": "INFO", "message":"loaded module [x-pack-fleet]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.975Z", "log.level": "INFO", "message":"loaded module [x-pack-graph]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.975Z", "log.level": "INFO", "message":"loaded module [x-pack-identity-provider]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.975Z", "log.level": "INFO", "message":"loaded module [x-pack-ilm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.975Z", "log.level": "INFO", "message":"loaded module [x-pack-logstash]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.975Z", "log.level": "INFO", "message":"loaded module [x-pack-ml]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.975Z", "log.level": "INFO", "message":"loaded module [x-pack-monitoring]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.976Z", "log.level": "INFO", "message":"loaded module [x-pack-ql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.976Z", "log.level": "INFO", "message":"loaded module [x-pack-rollup]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.976Z", "log.level": "INFO", "message":"loaded module [x-pack-security]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.976Z", "log.level": "INFO", "message":"loaded module [x-pack-shutdown]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.976Z", "log.level": "INFO", "message":"loaded module [x-pack-sql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.976Z", "log.level": "INFO", "message":"loaded module [x-pack-stack]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.976Z", "log.level": "INFO", "message":"loaded module [x-pack-text-structure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.976Z", "log.level": "INFO", "message":"loaded module [x-pack-voting-only-node]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.976Z", "log.level": "INFO", "message":"loaded module [x-pack-watcher]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.976Z", "log.level": "INFO", "message":"loaded module [x-pack-write-load-forecaster]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:00.977Z", "log.level": "INFO", "message":"no plugins loaded", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:02.794Z", "log.level": "WARN", "message":"Jan 19, 2023 10:50:02 PM org.apache.lucene.store.MMapDirectory lookupProvider", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"stderr","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"@timestamp":"2023-01-19T22:50:02.794Z", "log.level": "WARN", "message":"WARNING: You are running with Java 19. To make full use of MMapDirectory, please pass '--enable-preview' to the Java command line.", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"stderr","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"}
  {"timestamp": "2023-01-19T22:50:07+00:00", "message": "readiness probe failed", "curl_rc": "7"}
  {"timestamp": "2023-01-19T22:50:12+00:00", "message": "readiness probe failed", "curl_rc": "7"}
  {"timestamp": "2023-01-19T22:50:17+00:00", "message": "readiness probe failed", "curl_rc": "7"}

[naemono]

@melvin-suter It would be incredibly hard to troubleshoot why Elasticsearch isn't functioning in your specific environment without much more details, but the one thing that does stand out is storageClassName: nfs-rwo. Are you using an NFS backend to run Elasticsearch? What storage system is backing this?

If you could try and start a test cluster with nothing that couldn't be shared publicly, and run https://github.com/elastic/eck-diagnostics against it, and upload them here, we could try and assist further.

[naemono]

Also look through these old issues that could be relevant:

https://github.com/elastic/cloud-on-k8s/issues/3877 https://github.com/elastic/cloud-on-k8s/issues/3701 https://github.com/elastic/cloud-on-k8s/issues/3293

Do any of the initContainers show any errors?

[barkbay]

A few things odd in the logs:

• Skipping security auto configuration because the configuration file [/usr/share/elasticsearch/config/elasticsearch.yml] is missing or is not a regular file which seems to suggest that there is no Elasticsearch configuration file.

• {"message":"WARNING: [...] To make full use of MMapDirectory [...]","elasticsearch.node.name":"elastic-es-default-0","elasticsearch.cluster.name":"elastic"} while node.store.allow_mmap is false in your manifest.

I am closing this due to inactivity. If you want to reopen, could you verify that the manifest you provided is the one actually deployed? Thanks.