search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.57k stars 24.62k forks source link

Elasticsearch cluster on AWS ECS with discovery-ec2 fails to start after version 8.1.3 (8.2.0 onward) #92613

Closed RaJiska closed 11 months ago

RaJiska commented 1 year ago

Elasticsearch Version

8.5.3

Installed Plugins

discovery-ec2

Java Version

bundled

OS Version

Docker - elasticsearch:8.5.3

Problem Description

Past version 8.1.3, clustering on AWS cloud with plugin discovery-ec2 results in having the startup freezing right after the discovery-ec2 module is loaded, and IMDS call is made to retrieve the availability zone the instance is running on. The issue happens through my own image (details provided in steps), as well as through the original image with entrypoint overridden, plugin install, and manual start of the app.

Steps to Reproduce

Create a Docker image with the following files:

Dockerfile

FROM elasticsearch:8.5.3

ENV CLUSTER_NAME default
ENV EC2_INSTANCE_ID undefined

COPY elasticsearch.yml /usr/share/elasticsearch/config/
COPY --chown=elasticsearch:elasticsearch entrypoint.sh /usr/local/bin/new-entrypoint.sh
RUN chmod +x /usr/local/bin/new-entrypoint.sh

RUN echo "y" |bin/elasticsearch-plugin install discovery-ec2

ENTRYPOINT ["/bin/bash", "/usr/local/bin/new-entrypoint.sh"]

new-entrypoint.sh

#!/bin/bash

export EC2_INSTANCE_ID=$(curl --silent http://169.254.169.254/latest/meta-data/instance-id)

if [[ ! -e "$B64_TLS_CERTIFICATE" ]]; then
    echo -n "$B64_TLS_CERTIFICATE" |base64 -d > /usr/share/elasticsearch/config/elastic-certificates.p12
fi

# Resume to base image entrypoint
/bin/tini -s -- /usr/local/bin/docker-entrypoint.sh "$@"

elasticsearch.yml

cluster.name: ${CLUSTER_NAME}
node.name: ${EC2_INSTANCE_ID}
network.host: _ec2_

discovery.seed_providers: ec2
discovery.ec2.host_type: private_ip
discovery.ec2.any_group: true
discovery.ec2.endpoint: ec2.ap-southeast-1.amazonaws.com
cluster.initial_master_nodes: [i-0595b8xxxx, i-07fa13axxxx, i-0a5c21xxxx]

cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone

http.port: 9200
transport.tcp.port: 9300

node.data: true
node.master: true

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
logger.org.elasticsearch.discovery: DEBUG

You can run the image without any environment variable or extra configuration; whether the certificate is passed or not the result is the same as the start of the application will just hang after IMDS call (according to logs).

Logs (if relevant)

Created elasticsearch keystore in /usr/share/elasticsearch/config/elasticsearch.keystore
{"@timestamp":"2022-12-30T10:05:36.484Z", "log.level": "INFO", "message":"version[8.5.3], pid[175], build[docker/4ed5ee9afac63de92ec98f404ccbed7d3ba9584e/2022-12-05T18:22:22.226119656Z], OS[Linux/4.14.301-224.520.amzn2.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/19.0.1/19.0.1+10-21]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:36.490Z", "log.level": "INFO", "message":"JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:36.491Z", "log.level": "INFO", "message":"JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true,-Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-13108333766483741156, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1750m, -Xmx1750m, -XX:MaxDirectMemorySize=917504000, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:39.010Z", "log.level": "INFO", "message":"Package versions: jackson-annotations=2.13.2, jackson-core=2.13.2, jackson-databind=2.13.2.2, jackson-dataformat-xml=2.13.2, jackson-datatype-jsr310=2.13.2, azure-core=1.27.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"com.azure.core.implementation.jackson.JacksonVersion","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.062Z", "log.level":"DEBUG", "message":"Using either environment variables, system properties or instance profile credentials", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.discovery.ec2.Ec2ClientSettings","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.144Z", "log.level": "INFO", "message":"loaded module [aggs-matrix-stats]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.144Z", "log.level": "INFO", "message":"loaded module [analysis-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.145Z", "log.level": "INFO", "message":"loaded module [apm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.145Z", "log.level": "INFO", "message":"loaded module [constant-keyword]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.146Z", "log.level": "INFO", "message":"loaded module [data-streams]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.146Z", "log.level": "INFO", "message":"loaded module [frozen-indices]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.146Z", "log.level": "INFO", "message":"loaded module [ingest-attachment]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.146Z", "log.level": "INFO", "message":"loaded module [ingest-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.148Z", "log.level": "INFO", "message":"loaded module [ingest-geoip]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.148Z", "log.level": "INFO", "message":"loaded module [ingest-user-agent]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.148Z", "log.level": "INFO", "message":"loaded module [kibana]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.149Z", "log.level": "INFO", "message":"loaded module [lang-expression]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.149Z", "log.level": "INFO", "message":"loaded module [lang-mustache]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.149Z", "log.level": "INFO", "message":"loaded module [lang-painless]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.154Z", "log.level": "INFO", "message":"loaded module [legacy-geo]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.154Z", "log.level": "INFO", "message":"loaded module [mapper-extras]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.155Z", "log.level": "INFO", "message":"loaded module [mapper-version]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.155Z", "log.level": "INFO", "message":"loaded module [old-lucene-versions]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.156Z", "log.level": "INFO", "message":"loaded module [parent-join]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.156Z", "log.level": "INFO", "message":"loaded module [percolator]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.156Z", "log.level": "INFO", "message":"loaded module [rank-eval]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.157Z", "log.level": "INFO", "message":"loaded module [reindex]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.157Z", "log.level": "INFO", "message":"loaded module [repositories-metering-api]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.157Z", "log.level": "INFO", "message":"loaded module [repository-azure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.157Z", "log.level": "INFO", "message":"loaded module [repository-encrypted]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.158Z", "log.level": "INFO", "message":"loaded module [repository-gcs]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.158Z", "log.level": "INFO", "message":"loaded module [repository-s3]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.158Z", "log.level": "INFO", "message":"loaded module [repository-url]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.159Z", "log.level": "INFO", "message":"loaded module [runtime-fields-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.159Z", "log.level": "INFO", "message":"loaded module [search-business-rules]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.159Z", "log.level": "INFO", "message":"loaded module [searchable-snapshots]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.159Z", "log.level": "INFO", "message":"loaded module [snapshot-based-recoveries]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.160Z", "log.level": "INFO", "message":"loaded module [snapshot-repo-test-kit]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.160Z", "log.level": "INFO", "message":"loaded module [spatial]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.160Z", "log.level": "INFO", "message":"loaded module [transform]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.161Z", "log.level": "INFO", "message":"loaded module [transport-netty4]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.161Z", "log.level": "INFO", "message":"loaded module [unsigned-long]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.161Z", "log.level": "INFO", "message":"loaded module [vector-tile]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.162Z", "log.level": "INFO", "message":"loaded module [wildcard]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.162Z", "log.level": "INFO", "message":"loaded module [x-pack-aggregate-metric]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.163Z", "log.level": "INFO", "message":"loaded module [x-pack-analytics]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.163Z", "log.level": "INFO", "message":"loaded module [x-pack-async]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.163Z", "log.level": "INFO", "message":"loaded module [x-pack-async-search]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.164Z", "log.level": "INFO", "message":"loaded module [x-pack-autoscaling]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.164Z", "log.level": "INFO", "message":"loaded module [x-pack-ccr]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.164Z", "log.level": "INFO", "message":"loaded module [x-pack-core]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.165Z", "log.level": "INFO", "message":"loaded module [x-pack-deprecation]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.165Z", "log.level": "INFO", "message":"loaded module [x-pack-enrich]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.165Z", "log.level": "INFO", "message":"loaded module [x-pack-eql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.166Z", "log.level": "INFO", "message":"loaded module [x-pack-fleet]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.166Z", "log.level": "INFO", "message":"loaded module [x-pack-graph]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.166Z", "log.level": "INFO", "message":"loaded module [x-pack-identity-provider]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.166Z", "log.level": "INFO", "message":"loaded module [x-pack-ilm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.167Z", "log.level": "INFO", "message":"loaded module [x-pack-logstash]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.167Z", "log.level": "INFO", "message":"loaded module [x-pack-ml]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.167Z", "log.level": "INFO", "message":"loaded module [x-pack-monitoring]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.168Z", "log.level": "INFO", "message":"loaded module [x-pack-ql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.168Z", "log.level": "INFO", "message":"loaded module [x-pack-rollup]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.169Z", "log.level": "INFO", "message":"loaded module [x-pack-security]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.170Z", "log.level": "INFO", "message":"loaded module [x-pack-shutdown]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.171Z", "log.level": "INFO", "message":"loaded module [x-pack-sql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.171Z", "log.level": "INFO", "message":"loaded module [x-pack-stack]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.171Z", "log.level": "INFO", "message":"loaded module [x-pack-text-structure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.172Z", "log.level": "INFO", "message":"loaded module [x-pack-voting-only-node]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.172Z", "log.level": "INFO", "message":"loaded module [x-pack-watcher]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.172Z", "log.level": "INFO", "message":"loaded plugin [discovery-ec2]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
{"@timestamp":"2022-12-30T10:05:41.215Z", "log.level":"DEBUG", "message":"obtaining ec2 [placement/availability-zone] from ec2 meta-data url http://169.254.169.254/latest/meta-data/placement/availability-zone", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin","elasticsearch.node.name":"i-0595b8xxxx","elasticsearch.cluster.name":"monitoring"}
DaveCTurner commented 1 year ago

Could you capture and share here a thread dump captured with jstack, taken when the node appears to be stuck?

RaJiska commented 1 year ago

Hi @DaveCTurner ,

Just spin up a new EC2 and did all the setup from scratch with all the commands used to give you more visibility. I run the jstack commad as well. Following the results:

Terminal 1 (Build & Run):

[root@ip-172-1-35-235 ~]# ls
Dockerfile  elasticsearch.yml  entrypoint.sh
[root@ip-172-1-35-235 ~]# cat Dockerfile
FROM elasticsearch:8.5.3

ENV CLUSTER_NAME default
ENV EC2_INSTANCE_ID undefined

COPY elasticsearch.yml /usr/share/elasticsearch/config/
COPY --chown=elasticsearch:elasticsearch entrypoint.sh /usr/local/bin/new-entrypoint.sh
RUN chmod +x /usr/local/bin/new-entrypoint.sh

RUN echo "y" |bin/elasticsearch-plugin install discovery-ec2

ENTRYPOINT ["/bin/bash", "/usr/local/bin/new-entrypoint.sh"]
[root@ip-172-1-35-235 ~]# cat entrypoint.sh
#!/bin/bash

export EC2_INSTANCE_ID=$(curl --silent http://169.254.169.254/latest/meta-data/instance-id)

if [[ ! -e "$B64_TLS_CERTIFICATE" ]]; then
    echo -n "$B64_TLS_CERTIFICATE" |base64 -d > /usr/share/elasticsearch/config/elastic-certificates.p12
fi

# Resume to base image entrypoint
/bin/tini -s -- /usr/local/bin/docker-entrypoint.sh "$@"

[root@ip-172-1-35-235 ~]# cat elasticsearch.yml
cluster.name: ${CLUSTER_NAME}
node.name: ${EC2_INSTANCE_ID}
network.host: _ec2_

discovery.seed_providers: ec2
discovery.ec2.host_type: private_ip
discovery.ec2.any_group: true
discovery.ec2.endpoint: ec2.ap-southeast-1.amazonaws.com
cluster.initial_master_nodes: [i-0595b8xxxx, i-07fa13axxxx, i-0a5c21xxxx]

cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone

http.port: 9200
transport.tcp.port: 9300

node.data: true
node.master: true

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
logger.org.elasticsearch.discovery: DEBUG

[root@ip-172-1-35-235 ~]# docker build --no-cache -t elasticsearch-test .
Sending build context to Docker daemon  15.87kB
Step 1/8 : FROM elasticsearch:8.5.3
 ---> 7eac6adf7120
Step 2/8 : ENV CLUSTER_NAME default
 ---> Running in 69b40fe98adf
Removing intermediate container 69b40fe98adf
 ---> 397277455761
Step 3/8 : ENV EC2_INSTANCE_ID undefined
 ---> Running in 0fff7a8049c6
Removing intermediate container 0fff7a8049c6
 ---> 6f3d0b3fb0f9
Step 4/8 : COPY elasticsearch.yml /usr/share/elasticsearch/config/
 ---> 909df3fcfc56
Step 5/8 : COPY --chown=elasticsearch:elasticsearch entrypoint.sh /usr/local/bin/new-entrypoint.sh
 ---> 2bacfe23b246
Step 6/8 : RUN chmod +x /usr/local/bin/new-entrypoint.sh
 ---> Running in 14f2cbe38424
Removing intermediate container 14f2cbe38424
 ---> 43ce808800e0
Step 7/8 : RUN echo "y" |bin/elasticsearch-plugin install discovery-ec2
 ---> Running in 3442d4aa3419
-> Installing discovery-ec2
-> Downloading discovery-ec2 from elastic
[=================================================] 100%??
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.SocketPermission * connect,resolve
See https://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]-> Installed discovery-ec2
-> Please restart Elasticsearch to activate any plugins installed
Removing intermediate container 3442d4aa3419
 ---> ee7d437dfbb3
Step 8/8 : ENTRYPOINT ["/bin/bash", "/usr/local/bin/new-entrypoint.sh"]
 ---> Running in 186a56c90b41
Removing intermediate container 186a56c90b41
 ---> 5cc35132f41b
Successfully built 5cc35132f41b
Successfully tagged elasticsearch-test:latest
[root@ip-172-1-35-235 ~]# docker run --rm --name test elasticsearch-test
{"@timestamp":"2022-12-31T08:04:42.880Z", "log.level": "INFO", "message":"version[8.5.3], pid[71], build[docker/4ed5ee9afac63de92ec98f404ccbed7d3ba9584e/2022-12-05T18:22:22.226119656Z], OS[Linux/5.10.157-139.675.amzn2.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/19.0.1/19.0.1+10-21]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:42.894Z", "log.level": "INFO", "message":"JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:42.895Z", "log.level": "INFO", "message":"JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true,-Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-12201745460231581542, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1931m, -Xmx1931m, -XX:MaxDirectMemorySize=1012924416, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:45.519Z", "log.level": "INFO", "message":"Package versions: jackson-annotations=2.13.2, jackson-core=2.13.2, jackson-databind=2.13.2.2, jackson-dataformat-xml=2.13.2, jackson-datatype-jsr310=2.13.2, azure-core=1.27.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"com.azure.core.implementation.jackson.JacksonVersion","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.509Z", "log.level":"DEBUG", "message":"Using either environment variables, system properties or instance profile credentials", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.discovery.ec2.Ec2ClientSettings","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.584Z", "log.level": "INFO", "message":"loaded module [aggs-matrix-stats]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.585Z", "log.level": "INFO", "message":"loaded module [analysis-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.585Z", "log.level": "INFO", "message":"loaded module [apm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.585Z", "log.level": "INFO", "message":"loaded module [constant-keyword]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.585Z", "log.level": "INFO", "message":"loaded module [data-streams]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.586Z", "log.level": "INFO", "message":"loaded module [frozen-indices]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.586Z", "log.level": "INFO", "message":"loaded module [ingest-attachment]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.586Z", "log.level": "INFO", "message":"loaded module [ingest-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.586Z", "log.level": "INFO", "message":"loaded module [ingest-geoip]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.587Z", "log.level": "INFO", "message":"loaded module [ingest-user-agent]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.587Z", "log.level": "INFO", "message":"loaded module [kibana]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.587Z", "log.level": "INFO", "message":"loaded module [lang-expression]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.587Z", "log.level": "INFO", "message":"loaded module [lang-mustache]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.588Z", "log.level": "INFO", "message":"loaded module [lang-painless]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.588Z", "log.level": "INFO", "message":"loaded module [legacy-geo]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.588Z", "log.level": "INFO", "message":"loaded module [mapper-extras]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.588Z", "log.level": "INFO", "message":"loaded module [mapper-version]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.589Z", "log.level": "INFO", "message":"loaded module [old-lucene-versions]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.589Z", "log.level": "INFO", "message":"loaded module [parent-join]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.589Z", "log.level": "INFO", "message":"loaded module [percolator]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.590Z", "log.level": "INFO", "message":"loaded module [rank-eval]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.590Z", "log.level": "INFO", "message":"loaded module [reindex]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.590Z", "log.level": "INFO", "message":"loaded module [repositories-metering-api]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.590Z", "log.level": "INFO", "message":"loaded module [repository-azure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.591Z", "log.level": "INFO", "message":"loaded module [repository-encrypted]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.591Z", "log.level": "INFO", "message":"loaded module [repository-gcs]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.591Z", "log.level": "INFO", "message":"loaded module [repository-s3]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.591Z", "log.level": "INFO", "message":"loaded module [repository-url]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.592Z", "log.level": "INFO", "message":"loaded module [runtime-fields-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.592Z", "log.level": "INFO", "message":"loaded module [search-business-rules]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.592Z", "log.level": "INFO", "message":"loaded module [searchable-snapshots]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.592Z", "log.level": "INFO", "message":"loaded module [snapshot-based-recoveries]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.592Z", "log.level": "INFO", "message":"loaded module [snapshot-repo-test-kit]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.593Z", "log.level": "INFO", "message":"loaded module [spatial]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.593Z", "log.level": "INFO", "message":"loaded module [transform]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.593Z", "log.level": "INFO", "message":"loaded module [transport-netty4]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.593Z", "log.level": "INFO", "message":"loaded module [unsigned-long]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.594Z", "log.level": "INFO", "message":"loaded module [vector-tile]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.594Z", "log.level": "INFO", "message":"loaded module [wildcard]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.594Z", "log.level": "INFO", "message":"loaded module [x-pack-aggregate-metric]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.594Z", "log.level": "INFO", "message":"loaded module [x-pack-analytics]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.595Z", "log.level": "INFO", "message":"loaded module [x-pack-async]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.595Z", "log.level": "INFO", "message":"loaded module [x-pack-async-search]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.595Z", "log.level": "INFO", "message":"loaded module [x-pack-autoscaling]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.595Z", "log.level": "INFO", "message":"loaded module [x-pack-ccr]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.596Z", "log.level": "INFO", "message":"loaded module [x-pack-core]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.596Z", "log.level": "INFO", "message":"loaded module [x-pack-deprecation]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.596Z", "log.level": "INFO", "message":"loaded module [x-pack-enrich]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.596Z", "log.level": "INFO", "message":"loaded module [x-pack-eql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.597Z", "log.level": "INFO", "message":"loaded module [x-pack-fleet]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.597Z", "log.level": "INFO", "message":"loaded module [x-pack-graph]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.597Z", "log.level": "INFO", "message":"loaded module [x-pack-identity-provider]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.597Z", "log.level": "INFO", "message":"loaded module [x-pack-ilm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.597Z", "log.level": "INFO", "message":"loaded module [x-pack-logstash]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.598Z", "log.level": "INFO", "message":"loaded module [x-pack-ml]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.598Z", "log.level": "INFO", "message":"loaded module [x-pack-monitoring]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.598Z", "log.level": "INFO", "message":"loaded module [x-pack-ql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.598Z", "log.level": "INFO", "message":"loaded module [x-pack-rollup]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.599Z", "log.level": "INFO", "message":"loaded module [x-pack-security]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.599Z", "log.level": "INFO", "message":"loaded module [x-pack-shutdown]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.599Z", "log.level": "INFO", "message":"loaded module [x-pack-sql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.599Z", "log.level": "INFO", "message":"loaded module [x-pack-stack]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.600Z", "log.level": "INFO", "message":"loaded module [x-pack-text-structure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.600Z", "log.level": "INFO", "message":"loaded module [x-pack-voting-only-node]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.600Z", "log.level": "INFO", "message":"loaded module [x-pack-watcher]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.601Z", "log.level": "INFO", "message":"loaded plugin [discovery-ec2]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}
{"@timestamp":"2022-12-31T08:04:47.647Z", "log.level":"DEBUG", "message":"obtaining ec2 [placement/availability-zone] from ec2 meta-data url http://169.254.169.254/latest/meta-data/placement/availability-zone", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin","elasticsearch.node.name":"i-04d0373c282f6ad24","elasticsearch.cluster.name":"default"}

Terminal 2 (Running Jstack):

root@ip-172-1-35-235 bin]# docker exec -it test bash
elasticsearch@6d7e9c197bcd:~$ ./jdk/bin/jps
71 Elasticsearch
11 CliToolLauncher
107 Jps
elasticsearch@6d7e9c197bcd:~$ ./jdk/bin/jstack 71
2022-12-31 08:05:42
Full thread dump OpenJDK 64-Bit Server VM (19.0.1+10-21 mixed mode, sharing):

Threads class SMR info:
_java_thread_list=0x00007fea5c002040, length=15, elements={
0x00007feaa8029b20, 0x00007feaa8241cc0, 0x00007feaa82431a0, 0x00007feaa82447e0,
0x00007feaa8245c60, 0x00007feaa82470d0, 0x00007feaa8248ae0, 0x00007feaa824a070,
0x00007feaa824b540, 0x00007feaa8259d60, 0x00007feaa82c5200, 0x00007feaa8864900,
0x00007feaa8865f70, 0x00007feaa8866bd0, 0x00007fea5c0015d0
}

"main" #1 [73] prio=5 os_prio=0 cpu=6500.61ms elapsed=63.15s tid=0x00007feaa8029b20 nid=73 runnable  [0x00007feaaf823000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.Net.poll(java.base@19.0.1/Native Method)
        at sun.nio.ch.NioSocketImpl.park(java.base@19.0.1/NioSocketImpl.java:186)
        at sun.nio.ch.NioSocketImpl.park(java.base@19.0.1/NioSocketImpl.java:196)
        at sun.nio.ch.NioSocketImpl.implRead(java.base@19.0.1/NioSocketImpl.java:304)
        at sun.nio.ch.NioSocketImpl.read(java.base@19.0.1/NioSocketImpl.java:340)
        at sun.nio.ch.NioSocketImpl$1.read(java.base@19.0.1/NioSocketImpl.java:789)
        at java.net.Socket$SocketInputStream.read(java.base@19.0.1/Socket.java:1025)
        at java.io.BufferedInputStream.fill(java.base@19.0.1/BufferedInputStream.java:255)
        at java.io.BufferedInputStream.read1(java.base@19.0.1/BufferedInputStream.java:310)
        at java.io.BufferedInputStream.implRead(java.base@19.0.1/BufferedInputStream.java:382)
        at java.io.BufferedInputStream.read(java.base@19.0.1/BufferedInputStream.java:361)
        at sun.net.www.http.HttpClient.parseHTTPHeader(java.base@19.0.1/HttpClient.java:827)
        at sun.net.www.http.HttpClient.parseHTTP(java.base@19.0.1/HttpClient.java:759)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(java.base@19.0.1/HttpURLConnection.java:1684)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(java.base@19.0.1/HttpURLConnection.java:1585)
        at org.elasticsearch.discovery.ec2.AwsEc2Utils.lambda$getMetadataToken$1(AwsEc2Utils.java:49)
        at org.elasticsearch.discovery.ec2.AwsEc2Utils$$Lambda$844/0x00000008014eb4c8.run(Unknown Source)
        at java.security.AccessController.executePrivileged(java.base@19.0.1/AccessController.java:776)
        at java.security.AccessController.doPrivileged(java.base@19.0.1/AccessController.java:318)
        at org.elasticsearch.discovery.ec2.SocketAccess.doPrivileged(SocketAccess.java:31)
        at org.elasticsearch.discovery.ec2.AwsEc2Utils.getMetadataToken(AwsEc2Utils.java:37)
        at org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin.getAvailabilityZoneNodeAttributes(Ec2DiscoveryPlugin.java:147)
        at org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin.additionalSettings(Ec2DiscoveryPlugin.java:128)
        at org.elasticsearch.node.Node.mergePluginSettings(org.elasticsearch.server@8.5.3/Node.java:1635)
        at org.elasticsearch.node.Node.<init>(org.elasticsearch.server@8.5.3/Node.java:412)
        at org.elasticsearch.node.Node.<init>(org.elasticsearch.server@8.5.3/Node.java:318)
        at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(org.elasticsearch.server@8.5.3/Elasticsearch.java:214)
        at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(org.elasticsearch.server@8.5.3/Elasticsearch.java:214)
        at org.elasticsearch.bootstrap.Elasticsearch.main(org.elasticsearch.server@8.5.3/Elasticsearch.java:67)

"Reference Handler" #9 [81] daemon prio=10 os_prio=0 cpu=2.19ms elapsed=62.34s tid=0x00007feaa8241cc0 nid=81 waiting on condition  [0x00007fea8cf70000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.ref.Reference.waitForReferencePendingList(java.base@19.0.1/Native Method)
        at java.lang.ref.Reference.processPendingReferences(java.base@19.0.1/Reference.java:245)
        at java.lang.ref.Reference$ReferenceHandler.run(java.base@19.0.1/Reference.java:207)

"Finalizer" #10 [82] daemon prio=8 os_prio=0 cpu=0.34ms elapsed=62.34s tid=0x00007feaa82431a0 nid=82 in Object.wait()  [0x00007fea8ce6f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait0(java.base@19.0.1/Native Method)
        - waiting on <0x00000000875861d0> (a java.lang.ref.NativeReferenceQueue$Lock)
        at java.lang.Object.wait(java.base@19.0.1/Object.java:366)
        at java.lang.Object.wait(java.base@19.0.1/Object.java:339)
        at java.lang.ref.NativeReferenceQueue.await(java.base@19.0.1/NativeReferenceQueue.java:48)
        at java.lang.ref.ReferenceQueue.remove0(java.base@19.0.1/ReferenceQueue.java:158)
        at java.lang.ref.NativeReferenceQueue.remove(java.base@19.0.1/NativeReferenceQueue.java:89)
        - locked <0x00000000875861d0> (a java.lang.ref.NativeReferenceQueue$Lock)
        at java.lang.ref.Finalizer$FinalizerThread.run(java.base@19.0.1/Finalizer.java:173)

"Signal Dispatcher" #11 [83] daemon prio=9 os_prio=0 cpu=0.39ms elapsed=62.34s tid=0x00007feaa82447e0 nid=83 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #12 [84] daemon prio=9 os_prio=0 cpu=5.53ms elapsed=62.34s tid=0x00007feaa8245c60 nid=84 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Monitor Deflation Thread" #13 [85] daemon prio=9 os_prio=0 cpu=1.84ms elapsed=62.34s tid=0x00007feaa82470d0 nid=85 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #14 [86] daemon prio=9 os_prio=0 cpu=6667.54ms elapsed=62.34s tid=0x00007feaa8248ae0 nid=86 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #15 [87] daemon prio=9 os_prio=0 cpu=1927.88ms elapsed=62.34s tid=0x00007feaa824a070 nid=87 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Sweeper thread" #16 [88] daemon prio=9 os_prio=0 cpu=30.34ms elapsed=62.34s tid=0x00007feaa824b540 nid=88 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #17 [89] daemon prio=8 os_prio=0 cpu=2.04ms elapsed=62.32s tid=0x00007feaa8259d60 nid=89 waiting on condition  [0x00007fea8c768000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@19.0.1/Native Method)
        - parking to wait for  <0x0000000087586330> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@19.0.1/LockSupport.java:269)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@19.0.1/AbstractQueuedSynchronizer.java:1759)
        at java.lang.ref.ReferenceQueue.await(java.base@19.0.1/ReferenceQueue.java:71)
        at java.lang.ref.ReferenceQueue.remove0(java.base@19.0.1/ReferenceQueue.java:143)
        at java.lang.ref.ReferenceQueue.remove(java.base@19.0.1/ReferenceQueue.java:218)
        at jdk.internal.ref.CleanerImpl.run(java.base@19.0.1/CleanerImpl.java:140)
        at java.lang.Thread.run(java.base@19.0.1/Thread.java:1589)
        at jdk.internal.misc.InnocuousThread.run(java.base@19.0.1/InnocuousThread.java:186)

"Notification Thread" #18 [90] daemon prio=9 os_prio=0 cpu=0.06ms elapsed=62.03s tid=0x00007feaa82c5200 nid=90 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"process reaper (pid 92)" #20 [93] daemon prio=10 os_prio=0 cpu=0.24ms elapsed=60.86s tid=0x00007feaa8864900 nid=93 runnable  [0x00007fea8c261000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.ProcessHandleImpl.waitForProcessExit0(java.base@19.0.1/Native Method)
        at java.lang.ProcessHandleImpl$1.run(java.base@19.0.1/ProcessHandleImpl.java:146)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@19.0.1/ThreadPoolExecutor.java:1144)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@19.0.1/ThreadPoolExecutor.java:642)
        at java.lang.Thread.run(java.base@19.0.1/Thread.java:1589)
        at jdk.internal.misc.InnocuousThread.run(java.base@19.0.1/InnocuousThread.java:186)

"x-pack-ml-controller-stdout-pump" #21 [94] prio=5 os_prio=0 cpu=0.28ms elapsed=60.86s tid=0x00007feaa8865f70 nid=94 runnable  [0x00007fea8c160000]
   java.lang.Thread.State: RUNNABLE
        at java.io.FileInputStream.readBytes(java.base@19.0.1/Native Method)
        at java.io.FileInputStream.read(java.base@19.0.1/FileInputStream.java:293)
        at java.io.BufferedInputStream.read1(java.base@19.0.1/BufferedInputStream.java:308)
        at java.io.BufferedInputStream.implRead(java.base@19.0.1/BufferedInputStream.java:382)
        at java.io.BufferedInputStream.read(java.base@19.0.1/BufferedInputStream.java:367)
        - locked <0x00000000875867c0> (a java.lang.ProcessImpl$ProcessPipeInputStream)
        at sun.nio.cs.StreamDecoder.readBytes(java.base@19.0.1/StreamDecoder.java:333)
        at sun.nio.cs.StreamDecoder.implRead(java.base@19.0.1/StreamDecoder.java:376)
        at sun.nio.cs.StreamDecoder.lockedRead(java.base@19.0.1/StreamDecoder.java:219)
        at sun.nio.cs.StreamDecoder.read(java.base@19.0.1/StreamDecoder.java:173)
        at java.io.InputStreamReader.read(java.base@19.0.1/InputStreamReader.java:189)
        at java.io.BufferedReader.fill(java.base@19.0.1/BufferedReader.java:161)
        at java.io.BufferedReader.implReadLine(java.base@19.0.1/BufferedReader.java:371)
        at java.io.BufferedReader.readLine(java.base@19.0.1/BufferedReader.java:348)
        at java.io.BufferedReader.readLine(java.base@19.0.1/BufferedReader.java:437)
        at org.elasticsearch.bootstrap.Spawner.lambda$startPumpThread$3(org.elasticsearch.server@8.5.3/Spawner.java:108)
        at org.elasticsearch.bootstrap.Spawner$$Lambda$206/0x000000080114afb0.run(org.elasticsearch.server@8.5.3/Unknown Source)
        at java.lang.Thread.run(java.base@19.0.1/Thread.java:1589)

"x-pack-ml-controller-stderr-pump" #22 [95] prio=5 os_prio=0 cpu=0.21ms elapsed=60.86s tid=0x00007feaa8866bd0 nid=95 runnable  [0x00007fea53ffe000]
   java.lang.Thread.State: RUNNABLE
        at java.io.FileInputStream.readBytes(java.base@19.0.1/Native Method)
        at java.io.FileInputStream.read(java.base@19.0.1/FileInputStream.java:293)
        at java.io.BufferedInputStream.read1(java.base@19.0.1/BufferedInputStream.java:308)
        at java.io.BufferedInputStream.implRead(java.base@19.0.1/BufferedInputStream.java:382)
        at java.io.BufferedInputStream.read(java.base@19.0.1/BufferedInputStream.java:367)
        - locked <0x000000008758cc50> (a java.lang.ProcessImpl$ProcessPipeInputStream)
        at sun.nio.cs.StreamDecoder.readBytes(java.base@19.0.1/StreamDecoder.java:333)
        at sun.nio.cs.StreamDecoder.implRead(java.base@19.0.1/StreamDecoder.java:376)
        at sun.nio.cs.StreamDecoder.lockedRead(java.base@19.0.1/StreamDecoder.java:219)
        at sun.nio.cs.StreamDecoder.read(java.base@19.0.1/StreamDecoder.java:173)
        at java.io.InputStreamReader.read(java.base@19.0.1/InputStreamReader.java:189)
        at java.io.BufferedReader.fill(java.base@19.0.1/BufferedReader.java:161)
        at java.io.BufferedReader.implReadLine(java.base@19.0.1/BufferedReader.java:371)
        at java.io.BufferedReader.readLine(java.base@19.0.1/BufferedReader.java:348)
        at java.io.BufferedReader.readLine(java.base@19.0.1/BufferedReader.java:437)
        at org.elasticsearch.bootstrap.Spawner.lambda$startPumpThread$3(org.elasticsearch.server@8.5.3/Spawner.java:108)
        at org.elasticsearch.bootstrap.Spawner$$Lambda$206/0x000000080114afb0.run(org.elasticsearch.server@8.5.3/Unknown Source)
        at java.lang.Thread.run(java.base@19.0.1/Thread.java:1589)

"Attach Listener" #26 [145] daemon prio=9 os_prio=0 cpu=0.30ms elapsed=0.10s tid=0x00007fea5c0015d0 nid=145 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"VM Thread" os_prio=0 cpu=12.68ms elapsed=62.35s tid=0x00007feaa823ab50 nid=80 runnable

"GC Thread#0" os_prio=0 cpu=827.78ms elapsed=63.15s tid=0x00007feaa806e3b0 nid=74 runnable

"GC Thread#1" os_prio=0 cpu=833.12ms elapsed=63.15s tid=0x00007feaa807dba0 nid=77 runnable

"G1 Main Marker" os_prio=0 cpu=1.16ms elapsed=63.15s tid=0x00007feaa8073590 nid=75 runnable

"G1 Conc#0" os_prio=0 cpu=71.55ms elapsed=63.15s tid=0x00007feaa8074440 nid=76 runnable

"G1 Refine#0" os_prio=0 cpu=0.08ms elapsed=62.36s tid=0x00007feaa8000e80 nid=78 runnable

"G1 Service" os_prio=0 cpu=11.49ms elapsed=62.36s tid=0x00007feaa8216d00 nid=79 runnable

"VM Periodic Task Thread" os_prio=0 cpu=40.06ms elapsed=62.03s tid=0x00007feaa802b4b0 nid=91 waiting on condition

JNI global refs: 31, weak refs: 45
DaveCTurner commented 1 year ago

Thanks. Are you using IMDSv1 or v2? https://github.com/elastic/elasticsearch/pull/84410 indicates that we try v2 first and fall back to v1 but it's possible the fallback is not working.

elasticsearchmachine commented 1 year ago

Pinging @elastic/es-distributed (Team:Distributed)

RaJiska commented 1 year ago

@DaveCTurner ,

With the stack dump, I believe I found the source of the issue. The issue lies in the IMDSv2 support which I believe has been made the de-facto default in Elasticsearch. With IMDSv2, AWS by default restricts the maximum number of hops to 1 for PUT requests to the metadata API (IMDS), resulting in timeout in my case when running through container.

Instance role can be retrieved through the metadata API, which is blocked to containers by default which I assume is mostly so that containers, which already have their task role, may not access the instance role. On AWS, in a containerized environment task role credentials can be retrieved through the endpoint 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI.

As I haven't been going through the code, the above is mostly assumption, assuming they are right, a way to mitigate it would be either to have a IMDSv1 fallback mechanism (which would not work for IMDSv2 enforced container environments), or support IMDSv2 for container through the task role API as mentioned above.

Relevant AWS documentation for IAM task roles may be found here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html

RaJiska commented 1 year ago

Thanks. Are you using IMDSv1 or v2? #84410 indicates that we try v2 first and fall back to v1 but it's possible the fallback is not working.

IMDSv2 is not enforced, leaving IMDSv1 enabled:

$ aws ec2 describe-instances
[...]
"MetadataOptions": {
    "State": "applied",
    "HttpTokens": "optional",
    "HttpPutResponseHopLimit": 1,
    "HttpEndpoint": "enabled",
    "HttpProtocolIpv6": "disabled",
    "InstanceMetadataTags": "disabled"
},
[...]
DaveCTurner commented 1 year ago

I think that sounds about right @RaJiska. The discovery-ec2 plugin is not really intended for use in ECS because ECS has native support for service discovery via DNS, which Elasticsearch can use without any additional plugins. That means I expect we won't be adding support for the task role API in this plugin, and I recommend you switch to using the native service discovery mechanism too. But if the IMDSv1 fallback is not working correctly then that's a bug to be fixed.

RaJiska commented 1 year ago

@DaveCTurner ,

Understood, in this case, it seems like the IMDSv1 fallback mechanism is not working as intended in it.

May I also add that even though this issues appears for me on ECS, it is not an ECS problem, but more of an EC2 problem as this issue would also impact a containers started on an EC2 machine not running in an ECS cluster.

DaveCTurner commented 1 year ago

this issue would also impact a containers started on an EC2 machine not running in an ECS cluster.

I will strengthen my previous statement:

The discovery-ec2 plugin is ~not really intended for use in ECS~ only really intended for use when running Elasticsearch directly on EC2.

If you run Elasticsearch in an environment (e.g. a container) configured so as to prevent it from accessing the APIs that would be accessible if it were running directly on EC2 then this plugin won't work, and that's deliberate.

RaJiska commented 1 year ago

Understood.

For ECS, regarding DNS discovery mechanism, the documentation does not precise which records are being queried. Is the SRV record type supported?

DaveCTurner commented 1 year ago

I wouldn't expect SRV records to work because the docs say "hosts specified as hostnames are resolved to IP addresses" and SRV records aren't used for this kind of resolution. I believe the specifics of the name resolution mechanism are under the control of your sysadmin (e.g. on Unix-like systems it delegates to the system's getaddrinfo()) but in practice in most modern systems it will use DNS A and/or AAAA records.

DaveCTurner commented 11 months ago

Nothing to do here, closing.