search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.55k stars 24.61k forks source link

Deprecated feature use in 7.x index fails upgrade to 8.x but cannot be downgraded. #84199

Closed DaveCTurner closed 2 years ago

DaveCTurner commented 2 years ago

Elasticsearch Version

8.x

Installed Plugins

N/A

Java Version

bundled

OS Version

N/A

Problem Description

In https://github.com/elastic/elasticsearch/issues/81326 we added some protection against starting up Elasticsearch 8.x where the cluster held indices created in 6.x, allowing the user to downgrade back to 7.x and address the problem.

A user reported a related but still-extant problem: they were using a deprecated feature in their 7.x indices and missed the deprecation log message so they couldn't start the 8.x node, but Elasticsearch didn't detect the problem until after rearranging the data path (https://github.com/elastic/elasticsearch/pull/42489) so they could not downgrade to fix it either.

Steps to Reproduce

Create an index in 7.x using a deprecated feature. In this case the deprecation warning was as follows:

Camel case format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead.

Try and upgrade to 8.x and observe that the node won't start.

Downgrade back to 7.x and observe that this node won't start either.

Logs (if relevant)

From the 8.x node when attempting to upgrade ``` kubectl -n mas logs elasticsearch-84b66fb459-gjpfc {"@timestamp":"2022-02-21T18:48:43.416Z", "log.level": "INFO", "message":"version[8.0.0], pid[7], build[default/docker/1b6a7ece17463df5ff54a3e1302d825889aa1161/2022-02-03T16:47:57.507843096Z], OS[Linux/5.4.0-91-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.1/17.0.1+12]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:43.418Z", "log.level": "INFO", "message":"JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:43.418Z", "log.level": "INFO", "message":"JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-18423929716915370894, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms31744m, -Xmx31744m, -XX:MaxDirectMemorySize=16642998272, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:44.369Z", "log.level": "WARN", "message":"SLF4J: Failed to load class \"org.slf4j.impl.StaticLoggerBinder\".", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"stderr","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:44.370Z", "log.level": "WARN", "message":"SLF4J: Defaulting to no-operation (NOP) logger implementation", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"stderr","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:44.370Z", "log.level": "WARN", "message":"SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.", "ecs.version": [some module loaded lines deleted, this is standard docker image] "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.019Z", "log.level": "INFO", "message":"loaded module [repositories-metering-api]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.019Z", "log.level": "INFO", "message":"loaded module [repository-azure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.020Z", "log.level": "INFO", "message":"loaded module [repository-encrypted]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.020Z", "log.level": "INFO", "message":"loaded module [repository-gcs]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.020Z", "log.level": "INFO", "message":"loaded module [repository-s3]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.020Z", "log.level": "INFO", "message":"loaded module [repository-url]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.020Z", "log.level": "INFO", "message":"loaded module [runtime-fields-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.021Z", "log.level": "INFO", "message":"loaded module [search-business-rules]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.021Z", "log.level": "INFO", "message":"loaded module [searchable-snapshots]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.021Z", "log.level": "INFO", "message":"loaded module [snapshot-based-recoveries]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.021Z", "log.level": "INFO", "message":"loaded module [snapshot-repo-test-kit]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.021Z", "log.level": "INFO", "message":"loaded module [spatial]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.021Z", "log.level": "INFO", "message":"loaded module [transform]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.021Z", "log.level": "INFO", "message":"loaded module [transport-netty4]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [unsigned-long]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [vector-tile]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [vectors]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [wildcard]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [x-pack-aggregate-metric]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [x-pack-analytics]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [x-pack-async]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [x-pack-async-search]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [x-pack-autoscaling]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-ccr]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-core]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-data-streams]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-deprecation]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-enrich]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-eql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-fleet]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-graph]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-identity-provider]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-ilm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.024Z", "log.level": "INFO", "message":"loaded module [x-pack-logstash]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.024Z", "log.level": "INFO", "message":"loaded module [x-pack-ml]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.024Z", "log.level": "INFO", "message":"loaded module [x-pack-monitoring]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.024Z", "log.level": "INFO", "message":"loaded module [x-pack-ql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.024Z", "log.level": "INFO", "message":"loaded module [x-pack-rollup]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.024Z", "log.level": "INFO", "message":"loaded module [x-pack-security]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.024Z", "log.level": "INFO", "message":"loaded module [x-pack-shutdown]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.024Z", "log.level": "INFO", "message":"loaded module [x-pack-sql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.025Z", "log.level": "INFO", "message":"loaded module [x-pack-stack]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.025Z", "log.level": "INFO", "message":"loaded module [x-pack-text-structure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.025Z", "log.level": "INFO", "message":"loaded module [x-pack-voting-only-node]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.025Z", "log.level": "INFO", "message":"loaded module [x-pack-watcher]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.025Z", "log.level": "INFO", "message":"no plugins loaded", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.045Z", "log.level": "INFO", "message":"using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/md2)]], net usable_space [188.9gb], net total_space [461gb], types [ext4]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.env.NodeEnvironment","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.045Z", "log.level": "INFO", "message":"heap size [31gb], compressed ordinary object pointers [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.env.NodeEnvironment","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:45.097Z", "log.level": "INFO", "message":"node name [elasticsearch-84b66fb459-gjpfc], node ID [jSFuEwKUR9q65j4Ta4E-cQ], cluster name [docker-cluster], roles [data_content, transform, data_hot, ml, data_frozen, ingest, data_cold, data, remote_cluster_client, master, data_warm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:48.398Z", "log.level": "INFO", "message":"Security is disabled", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.security.Security","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:48.728Z", "log.level": "INFO", "message":"creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.transport.netty4.NettyAllocator","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:48.763Z", "log.level": "INFO", "message":"using discovery type [single-node] and seed hosts providers [settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.discovery.DiscoveryModule","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:49.150Z", "log.level": "INFO", "message":"initialized", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:49.150Z", "log.level": "INFO", "message":"starting ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:49.161Z", "log.level": "INFO", "message":"persistent cache index loaded", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:49.161Z", "log.level": "INFO", "message":"deprecation component started", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.deprecation.logging.DeprecationIndexingComponent","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:49.212Z", "log.level": "INFO", "message":"publish_address {10.1.51.245:9300}, bound_addresses {[::]:9300}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.transport.TransportService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:49.286Z", "log.level": "WARN", "message":"Skipping unknown custom object with type system_index_migration", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.cluster.metadata.Metadata","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:49.364Z", "log.level":"ERROR", "message":"uncaught exception in thread [main]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.ElasticsearchUncaughtExceptionHandler","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster","error.type":"org.elasticsearch.bootstrap.StartupException","error.message":"java.lang.IllegalStateException: Failed to parse mappings for index [[reindexed-v7-messages1-2.4.0/x7yO_28nSVaRv08z3qnr1A]]","error.stack_trace":"java.lang.IllegalStateException: Failed to parse mappings for index [[reindexed-v7-messages1-2.4.0/x7yO_28nSVaRv08z3qnr1A]]\nLikely root cause: java.lang.IllegalArgumentException: Unknown pattern letter: t\n\tat java.base/java.time.format.DateTimeFormatterBuilder.parsePattern(DateTimeFormatterBuilder.java:1882)\n\tat java.base/java.time.format.DateTimeFormatterBuilder.appendPattern(DateTimeFormatterBuilder.java:1772)\n\tat org.elasticsearch.common.time.DateFormatters.forPattern(DateFormatters.java:2135)\n\tat org.elasticsearch.common.time.DateFormatter.forPattern(DateFormatter.java:116)\n\tat org.elasticsearch.index.mapper.DateFieldMapper$Builder.buildFormatter(DateFieldMapper.java:277)\n\tat org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:331)\n\tat org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:220)\n\tat org.elasticsearch.index.mapper.ObjectMapper$Builder.buildMappers(ObjectMapper.java:100)\n\tat org.elasticsearch.index.mapper.RootObjectMapper$Builder.build(RootObjectMapper.java:101)\n\tat org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:149)\n\tat org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:94)\n\tat org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:359)\n\tat org.elasticsearch.index.mapper.MapperService.mergeAndApplyMappings(MapperService.java:338)\n\tat org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:324)\n\tat org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:195)\n\tat org.elasticsearch.cluster.metadata.IndexMetadataVerifier.verifyIndexMetadata(IndexMetadataVerifier.java:91)\n\tat org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:226)\n\tat org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:213)\n\tat org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:127)\n\tat org.elasticsearch.node.Node.start(Node.java:1124)\n\tat org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:272)\n\tat org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:367)\n\tat org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166)\n\tat org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157)\n\tat org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)\n\tat org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)\n\tat org.elasticsearch.cli.Command.main(Command.java:77)\n\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)\n\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)\nFor complete error details, refer to the log at /usr/share/elasticsearch/logs/docker-cluster.log\n"} uncaught exception in thread [main] java.lang.IllegalStateException: Failed to parse mappings for index [[reindexed-v7-messages1-2.4.0/x7yO_28nSVaRv08z3qnr1A]] Likely root cause: java.lang.IllegalArgumentException: Unknown pattern letter: t at java.base/java.time.format.DateTimeFormatterBuilder.parsePattern(DateTimeFormatterBuilder.java:1882) at java.base/java.time.format.DateTimeFormatterBuilder.appendPattern(DateTimeFormatterBuilder.java:1772) at org.elasticsearch.common.time.DateFormatters.forPattern(DateFormatters.java:2135) at org.elasticsearch.common.time.DateFormatter.forPattern(DateFormatter.java:116) at org.elasticsearch.index.mapper.DateFieldMapper$Builder.buildFormatter(DateFieldMapper.java:277) at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:331) at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:220) at org.elasticsearch.index.mapper.ObjectMapper$Builder.buildMappers(ObjectMapper.java:100) at org.elasticsearch.index.mapper.RootObjectMapper$Builder.build(RootObjectMapper.java:101) at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:149) at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:94) at org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:359) at org.elasticsearch.index.mapper.MapperService.mergeAndApplyMappings(MapperService.java:338) at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:324) at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:195) at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.verifyIndexMetadata(IndexMetadataVerifier.java:91) at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:226) at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:213) at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:127) at org.elasticsearch.node.Node.start(Node.java:1124) at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:272) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:367) at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) at org.elasticsearch.cli.Command.main(Command.java:77) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) For complete error details, refer to the log at /usr/share/elasticsearch/logs/docker-cluster.log {"@timestamp":"2022-02-21T18:48:49.366Z", "log.level": "INFO", "message":"stopping ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-2","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:49.392Z", "log.level": "INFO", "message":"stopped", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-2","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:49.392Z", "log.level": "INFO", "message":"closing ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-2","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} {"@timestamp":"2022-02-21T18:48:49.397Z", "log.level": "INFO", "message":"closed", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-2","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"} ```
From the 7.x node when attempting to downgrade ``` $ kubectl -n mas logs elasticsearch-75d4bf46bb-h6xcp {"type": "server", "timestamp": "2022-02-21T18:45:46,138Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "version[7.17.0], pid[7], build[default/docker/bee86328705acaa9a6daede7140defd4d9ec56bd/2022-01-28T08:36:04.875279988Z], OS[Linux/5.4.0-91-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.1/17.0.1+12]" } {"type": "server", "timestamp": "2022-02-21T18:45:46,140Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" } {"type": "server", "timestamp": "2022-02-21T18:45:46,141Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-5817322145606499066, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms31744m, -Xmx31744m, -XX:MaxDirectMemorySize=16642998272, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,476Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [aggs-matrix-stats]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,476Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [analysis-common]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,476Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [constant-keyword]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [frozen-indices]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [ingest-common]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [ingest-geoip]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [ingest-user-agent]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [kibana]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [lang-expression]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [lang-mustache]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [lang-painless]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [legacy-geo]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [mapper-extras]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [mapper-version]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [parent-join]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [percolator]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [rank-eval]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [reindex]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [repositories-metering-api]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [repository-encrypted]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [repository-url]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [runtime-fields-common]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [search-business-rules]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [searchable-snapshots]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [snapshot-repo-test-kit]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [spatial]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [transform]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [transport-netty4]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [unsigned-long]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [vector-tile]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [vectors]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [wildcard]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-aggregate-metric]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-analytics]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-async]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-async-search]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-autoscaling]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-ccr]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-core]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-data-streams]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-deprecation]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-enrich]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-eql]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-fleet]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-graph]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-identity-provider]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-ilm]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-logstash]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-ml]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-monitoring]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-ql]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-rollup]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-security]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-shutdown]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-sql]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-stack]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-text-structure]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-voting-only-node]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,484Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-watcher]" } {"type": "server", "timestamp": "2022-02-21T18:45:47,484Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "no plugins loaded" } {"type": "server", "timestamp": "2022-02-21T18:45:47,495Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "uncaught exception in thread [main]", "stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: data path [/usr/share/elasticsearch/data] is not compatible with Elasticsearch v7.17.0, perhaps it has already been upgraded to a later version", "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.17.0.jar:7.17.0]", ```
elasticmachine commented 2 years ago

Pinging @elastic/es-core-infra (Team:Core/Infra)

weslambert commented 2 years ago

I am facing the same issue, and am looking forward to any updates! 😄

damn-simple-scripts commented 2 years ago

I was experiencing the same issue. as hotfix, I opened the _state (in my case /var/lib/elasticsearch/_state) in luke and deleted the broken indices. Make sure to use 9.0.0's luke, as can be found in apache archives since lucene91 is not supported by elastic. (warning from poster below: this may cause damage to your cluster and is unsafe)) The troublesome indices in my case were elastalert_status_error, elastalert_status_silence, elastalert_status_past, elastalert_status and elastalert_status_status.

I hope that this may help someone.

DaveCTurner commented 2 years ago

as hotfix, I opened the _state

This is likely to be harmful to your cluster and we very strongly recommend not to do this. See for instance these docs:

WARNING: Don’t modify anything within the data directory or run processes that might interfere with its contents. If something other than Elasticsearch modifies the contents of the data directory, then Elasticsearch may fail, reporting corruption or other data inconsistencies, or may appear to work correctly having silently lost some of your data.

You can do whatever you want to your own data of course, but it is irresponsible to suggest this path to anyone else.

upcFrost commented 2 years ago

Any solution for this? I don't remember having any critical warnings regarding this in the Update Assistant. For my case it would be ok even if this particular index would be deleted, doesn't really matter unless it will affect other indexes

Edit: Somehow managed to fix it. In my case I didn't need those broken indexes, they were leftovers from the previous config, so it was probably a bit easier than trying to fix everything. What I did was building elastic from source changing the place where the exception is thrown to just return null similar to the normal output. After that I've booted up all required nodes with this version, deleted faulty indexes, and restarted the stock version of elastic. Worked fine except that I used the master branch for the custom build and it updated the syntax to Lucene91, which kinda forced me to move to the snapshot from the stable build. Still better than having everything down

jgregmac commented 2 years ago

Any solution for this? I don't remember having any critical warnings regarding this in the Update Assistant. For my case it would be ok even if this particular index would be deleted, doesn't really matter unless it will affect other indexes

Edit: Somehow managed to fix it. In my case I didn't need those broken indexes, they were leftovers from the previous config, so it was probably a bit easier than trying to fix everything. What I did was building elastic from source changing the place where the exception is thrown to just return null similar to the normal output. After that I've booted up all required nodes with this version, deleted faulty indexes, and restarted the stock version of elastic. Worked fine except that I used the master branch for the custom build and it updated the syntax to Lucene91, which kinda forced me to move to the snapshot from the stable build. Still better than having everything down

This was really helpful. Thank you!

For reference, I checked out the v8.2.0 branch, then changed line 198-201 of the IndexMetadataVerifier.java file as follows:

from:

        } catch (Exception ex) {
            // Wrap the inner exception so we have the index name in the exception message
            throw new IllegalStateException("Failed to parse mappings for index [" + indexMetadata.getIndex() + "]", ex);
        }

to:

        } catch (Exception ignore) {
            // Wrap the inner exception so we have the index name in the exception message
            // throw new IllegalStateException("Failed to parse mappings for index [" + indexMetadata.getIndex() + "]", ex);
        }

setting the catch block to return null; did not work for me, as the gradle compiler did not like that.

The code then compiled cleanly, and I was able to build a docker image to run in my cluster, then delete the bad indices using "curl":

# Check on the unassigned shards:
$ curl -k -u $ELASTIC_USERNAME:$ELASTIC_PASSWORD https://localhost:9200/_cat/shards?h=index,shards,state,prirep,unassigned.reason | grep UNASSIGNED

elastalert_status_status                                      UNASSIGNED p ALLOCATION_FAILED
elastalert_status_status                                      UNASSIGNED r CLUSTER_RECOVERED
elastalert_status_error                                       UNASSIGNED p ALLOCATION_FAILED
elastalert_status_error                                       UNASSIGNED r CLUSTER_RECOVERED
elastalert_status                                             UNASSIGNED p ALLOCATION_FAILED
elastalert_status                                             UNASSIGNED r CLUSTER_RECOVERED
elastalert_status_past                                        UNASSIGNED p ALLOCATION_FAILED
elastalert_status_past                                        UNASSIGNED r CLUSTER_RECOVERED
elastalert_status_silence                                     UNASSIGNED p ALLOCATION_FAILED
elastalert_status_silence                                     UNASSIGNED r CLUSTER_RECOVERED

# If you are ready to delete all of the indices related to these shards, run:
curl -k -u $ELASTIC_USERNAME:$ELASTIC_PASSWORD https://localhost:9200/_cat/shards | \
    grep UNASSIGNED | awk {'print $1'} | \
    xargs -i curl -k -u $ELASTIC_USERNAME:$ELASTIC_PASSWORD -XDELETE "https://localhost:9200/{}"
kaisbaccour commented 2 years ago

Just sorted this out with compiling from sources for my 4VM cluster because I had elastalert deprecated indices

Here are the commands I executed to compile fix and go back to the official release

`git clone https://github.com/elastic/elasticsearch.git cd elasticsearch git checkout v8.2.0 vim server/src/main/java/org/elasticsearch/cluster/metadata/IndexMetadataVerifier.java ###########commment line 200 with double slash ############ change line 197 from ex to ignore wget https://download.java.net/java/GA/jdk17.0.2/dfd4a8d0985749f896bed50d7138ee7f/8/GPL/openjdk-17.0.2_linux-x64_bin.tar.gz tar xvf openjdk-17.0.2_linux-x64_bin.tar.gz sudo mv jdk-17.0.2/ /opt/jdk-17/ echo 'export JAVA_HOME=/opt/jdk-17' | sudo tee /etc/profile.d/java17.sh echo 'export PATH=$JAVA_HOME/bin:$PATH'|sudo tee -a /etc/profile.d/java17.sh source /etc/profile.d/java17.sh echo $JAVA_HOME

apt-get install -y unzip zip curl -s "https://get.sdkman.io" | bash bash sdk install gradle 7.4.2 ./gradlew localDistro ########### the binary should be in ./build/distribution/local/elasticsearch-8.2.0-SNAPSHOT

########compress your SNAPSHOT to distribute it across your cluster fleet cd ./build/distribution/local/ tar -czvf elasticsearch-8.2.0-SNAPSHOT.tar.gz elasticsearch-8.2.0-SNAPSHOT cp -R ./build/distribution/local/elasticsearch-8.2.0-SNAPSHOT /usr/share/elasticsearch-8.2.0-SNAPSHOT

scp the SNAPSHOT to all the servers.

##########make a local backup mv /usr/share/elasticsearch /usr/share/elasticsearch_backup #########untar cd /tmp tar -xvf elasticsearch-8.2.0-SNAPSHOT.tar.gz

cp -R /tmp/elasticsearch-8.2.0-SNAPSHOT/ /usr/share/elasticsearch/ #######cp -R build/distribution/local/elasticsearch-8.2.0-SNAPSHOT /usr/share/elasticsearch rm -rf /usr/share/elasticsearch/jdk/ cp -R /usr/share/elasticsearch_backup/jdk/ /usr/share/elasticsearch/jdk/

vim /etc/passwd --- /bin/bash to elasticsearch user

chmod +x /etc/default/elasticsearch su elasticsearch /etc/default/elasticsearch export ES_PATH_CONF=/etc/elasticsearch export PID_DIR=/var/run/elasticsearch export ES_SD_NOTIFY=true cd /usr/share/elasticsearch/ ./bin/elasticsearch

#########clean indices from UI

###########GO BACK TO THE ELASTIC RELEASE

go back toroot again

mv /usr/share/elasticsearch /usr/share/patch cp -R /usr/share/elasticsearch_backup /usr/share/elasticsearch systemctl restart elasticsearch systemctl status elasticsearch

vim /etc/passwd --- /bin/false to elasticsearch user `

yenonn commented 2 years ago

hi @jgregmac Do you have the patched docker image with you that I can use? I am currently facing the same problem as described here.

yenonn commented 2 years ago

Thanks, I have solved the problem with the pointers above. Thanks!

jgregmac commented 2 years ago

@yenonn , sorry for the delay in reply, but I am glad to hear that you were able to resolve your problem.

I did not keep my docker image, but I will note that it is easy enough to build your own (for a given value of "easy"). @kaisbaccour provides a good starting point. In addition to cloning the elasticsearch repo and installing the JDK and Gradle prerequisites, you also will need Docker (docker-ce) and have configured permissions for the current user to run docker without sudo. I was hoping that "podman" with the "podman-docker" packages would be adequate, but they were not.

Once all the prerequisites are in place, you cd to the elasticsearch repo root and simply run:

/gradlew :distribution:docker:docker-export:assemble

elasticsearch should compile and build a container image using the Dockerfile in the repo. The resulting container image can be accessed using docker image list and can be published using docker image push.

cawoodm commented 2 years ago

Anyone having proxy/ssl issues should add their proxy to the gradle.properties file like to:

systemProp.https.proxyHost=your-proxy
systemProp.https.proxyPort=8888
flomsk commented 2 years ago

I also faced that problem with elastalert indices when was upgrading Elastic stack from 7.17 to 8.x (running ECK k8s, with 4 replicas of elasticsearch)

Firstly, there is a small how-to about it on elastalert2 docs

Second, even I deleted indices before, one of my node was failing with Failed to parse mappings for index

My workaround was: I manually decreased Statefulset replicas count from 4 to 3, then ECK applied back 4 replicas and all of them green.

an-tex commented 1 year ago

Thanks to https://github.com/elastic/elasticsearch/pull/88914 you can now go straight from 7.17.7 to 8.5.0 without having this issue (you'll still have to recreate the indices as stated in https://elastalert2.readthedocs.io/en/latest/recipes/faq.html?highlight=create-index#does-elastalert-2-support-elasticsearch-8 )