Update cluster by ansible 2.13 to 2.17.1

[dzaczek]

Describe the bug Im trying to use ansible playbook to update opensearch 2.13 to to 2.17.1 on ubuntu 22.04 LTS 2.13 was installed also by playbook few month ago . Playbook failing during restart open search waiting , geeting timeout.

To Reproduce Steps to reproduce the behavior: Install version 2.13 by ansible playbook (iac_enable: false) Install version 2.17.1 by ansible playbook (iac_enable: true, copy_custom_security_configs: true) )

Expected behavior Working able to start cluster

LOGS

[2024-10-02T09:47:29,489][ERROR][o.o.b.Bootstrap          ] [a4211-os-master-1] Exception
java.lang.IllegalStateException: jar hell!
class: org.apache.lucene.queries.CommonTermsQuery
jar1: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar
jar2: /usr/share/opensearch/lib/lucene-queries-9.11.1.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.17.1]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.17.1]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.13.0.jar:2.17.1]
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.13.0.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.17.1.jar:2.17.1]
[2024-10-02T09:47:29,497][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [a4211-os-master-1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!
class: org.apache.lucene.queries.CommonTermsQuery
jar1: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar
jar2: /usr/share/opensearch/lib/lucene-queries-9.11.1.jar
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.13.0.jar:2.17.1]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.13.0.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.17.1.jar:2.17.1]
Caused by: java.lang.IllegalStateException: jar hell!
class: org.apache.lucene.queries.CommonTermsQuery
jar1: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar
jar2: /usr/share/opensearch/lib/lucene-queries-9.11.1.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.17.1]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.17.1]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.17.1.jar:2.17.1]
        ... 6 more

gc log:

[2024-10-02T09:47:26.855+0000][171247][gc,init] CardTable entry size: 512
[2024-10-02T09:47:26.855+0000][171247][gc     ] Using G1
[2024-10-02T09:47:27.782+0000][171247][gc,init] Version: 21.0.4+7-LTS (release)
[2024-10-02T09:47:27.782+0000][171247][gc,init] CPUs: 2 total, 2 available
[2024-10-02T09:47:27.782+0000][171247][gc,init] Memory: 7937M
[2024-10-02T09:47:27.782+0000][171247][gc,init] Large Page Support: Disabled
[2024-10-02T09:47:27.782+0000][171247][gc,init] NUMA Support: Disabled
[2024-10-02T09:47:27.782+0000][171247][gc,init] Compressed Oops: Enabled (Zero based)
[2024-10-02T09:47:27.782+0000][171247][gc,init] Heap Region Size: 2M
[2024-10-02T09:47:27.782+0000][171247][gc,init] Heap Min Capacity: 4G
[2024-10-02T09:47:27.782+0000][171247][gc,init] Heap Initial Capacity: 4G
[2024-10-02T09:47:27.782+0000][171247][gc,init] Heap Max Capacity: 4G
[2024-10-02T09:47:27.782+0000][171247][gc,init] Pre-touch: Enabled
[2024-10-02T09:47:27.782+0000][171247][gc,init] Parallel Workers: 2
[2024-10-02T09:47:27.782+0000][171247][gc,init] Concurrent Workers: 1
[2024-10-02T09:47:27.782+0000][171247][gc,init] Concurrent Refinement Workers: 2
[2024-10-02T09:47:27.782+0000][171247][gc,init] Periodic GC: Disabled
[2024-10-02T09:47:27.790+0000][171247][gc,metaspace] CDS archive(s) mapped at: [0x00007f83cf000000-0x00007f83cfcae000-0x00007f83cfcae000), size 13295616, SharedBaseAddress: 0x00007f83cf000000, ArchiveRelocationMode: 1.
[2024-10-02T09:47:27.790+0000][171247][gc,metaspace] Compressed class space mapped at: 0x00007f83d0000000-0x00007f8410000000, reserved size: 1073741824
[2024-10-02T09:47:27.791+0000][171247][gc,metaspace] Narrow klass base: 0x00007f83cf000000, Narrow klass shift: 0, Narrow klass range: 0x100000000
[2024-10-02T09:47:28.254+0000][171247][safepoint   ] Safepoint "ICBufferFull", Time since last: 458384473 ns, Reaching safepoint: 3732 ns, Cleanup: 90564 ns, At safepoint: 1474 ns, Total: 95770 ns
[2024-10-02T09:47:28.831+0000][171247][safepoint   ] Safepoint "ICBufferFull", Time since last: 576749324 ns, Reaching safepoint: 3039 ns, Cleanup: 107701 ns, At safepoint: 1756 ns, Total: 112496 ns
[2024-10-02T09:47:29.507+0000][171247][gc,heap,exit] Heap
[2024-10-02T09:47:29.507+0000][171247][gc,heap,exit]  garbage-first heap   total 4194304K, used 60446K [0x0000000700000000, 0x0000000800000000)
[2024-10-02T09:47:29.507+0000][171247][gc,heap,exit]   region size 2048K, 29 young (59392K), 0 survivors (0K)
[2024-10-02T09:47:29.507+0000][171247][gc,heap,exit]  Metaspace       used 15368K, committed 15680K, reserved 1114112K
[2024-10-02T09:47:29.507+0000][171247][gc,heap,exit]   class space    used 1849K, committed 1984K, reserved 1048576K
~

Host/Environment (please complete the following information):

• Ansible Version: 2.15.0
• Playbook Version: 3c513cf ·

[dzaczek]

This same problem i have when im trying to update from 2.13 to 2.14

2024-10-02T12:54:37,183][ERROR][o.o.b.Bootstrap          ] [a4212-os-master-2] Exception
java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.14.0.jar:2.14.0]
[2024-10-02T12:54:37,190][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [a4212-os-master-2] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.14.0.jar:2.14.0]
Caused by: java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.14.0.jar:2.14.0]
        ... 6 more

[dzaczek]

Sorry i click by mistake close :)

[dzaczek]

Im trying to check in logs enabled debug logs

Oct 02 13:15:18 a4212-os-master-2 opensearch[207812]: class: org.opensearch.plugins.ExtendedPluginsClassLoader
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.14.0.jar:2.14.0]
        ... 6 more
[2024-10-02T13:17:31,214][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Looking in classpath from jdk.internal.loader.ClassLoaders$AppClassLoader@4f2410ac for /com/sun/jna/linux-x86-64/libjnidispatch.so
[2024-10-02T13:17:31,217][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Found library resource at jar:file:/usr/share/opensearch/lib/jna-5.13.0.jar!/com/sun/jna/linux-x86-64/libjnidispatch.so
[2024-10-02T13:17:31,219][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Extracting library to /usr/share/opensearch/.cache/JNA/temp/jna15514319990484742182.tmp
[2024-10-02T13:17:31,221][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Trying /usr/share/opensearch/.cache/JNA/temp/jna15514319990484742182.tmp
[2024-10-02T13:17:31,223][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Found jnidispatch at /usr/share/opensearch/.cache/JNA/temp/jna15514319990484742182.tmp
[2024-10-02T13:17:31,264][DEBUG][o.o.b.SystemCallFilter   ] [a4212-os-master-2] Linux seccomp filter installation successful, threads: [all]
[2024-10-02T13:17:31,680][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] java.class.path: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar:/usr/share/opensearch/lib/lucene-spatial-extras-9.10.0.jar:/usr/share/opensearch/lib/lucene-memory-9.10.0.jar:/usr/share/opensearch/lib/opensearch-cli-2.13.0.jar:/usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar:/usr/share/opensearch/lib/java-version-checker-2.14.0.jar:/usr/share/opensearch/lib/opensearch-compress-2.13.0.jar:/usr/share/opensearch/lib/opensearch-core-2.14.0.jar:/usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar:/usr/share/opensearch/lib/java-version-checker-2.13.0.jar:/usr/share/opensearch/lib/opensearch-2.14.0.jar:/usr/share/opensearch/lib/jna-5.13.0.jar:/usr/share/opensearch/lib/opensearch-core-2.13.0.jar:/usr/share/opensearch/lib/lucene-grouping-9.10.0.jar:/usr/share/opensearch/lib/opensearch-compress-2.14.0.jar:/usr/share/opensearch/lib/lucene-spatial3d-9.10.0.jar:/usr/share/opensearch/lib/lucene-analysis-common-9.10.0.jar:/usr/share/opensearch/lib/joda-time-2.12.2.jar:/usr/share/opensearch/lib/opensearch-launchers-2.13.0.jar:/usr/share/opensearch/lib/opensearch-secure-sm-2.14.0.jar:/usr/share/opensearch/lib/jakarta.annotation-api-1.3.5.jar:/usr/share/opensearch/lib/protobuf-java-3.22.3.jar:/usr/share/opensearch/lib/jts-core-1.15.0.jar:/usr/share/opensearch/lib/opensearch-cli-2.14.0.jar:/usr/share/opensearch/lib/jackson-dataformat-yaml-2.17.0.jar:/usr/share/opensearch/lib/lucene-join-9.10.0.jar:/usr/share/opensearch/lib/reactive-streams-1.0.4.jar:/usr/share/opensearch/lib/zstd-jni-1.5.5-5.jar:/usr/share/opensearch/lib/opensearch-geo-2.13.0.jar:/usr/share/opensearch/lib/spatial4j-0.7.jar:/usr/share/opensearch/lib/opensearch-common-2.13.0.jar:/usr/share/opensearch/lib/opensearch-launchers-2.14.0.jar:/usr/share/opensearch/lib/opensearch-x-content-2.13.0.jar:/usr/share/opensearch/lib/reactor-core-3.5.15.jar:/usr/share/opensearch/lib/jackson-core-2.17.0.jar:/usr/share/opensearch/lib/lucene-core-9.10.0.jar:/usr/share/opensearch/lib/lucene-queryparser-9.10.0.jar:/usr/share/opensearch/lib/opensearch-telemetry-2.13.0.jar:/usr/share/opensearch/lib/jzlib-1.1.3.jar:/usr/share/opensearch/lib/lucene-highlighter-9.10.0.jar:/usr/share/opensearch/lib/opensearch-secure-sm-2.13.0.jar:/usr/share/opensearch/lib/jopt-simple-5.0.4.jar:/usr/share/opensearch/lib/HdrHistogram-2.1.12.jar:/usr/share/opensearch/lib/opensearch-telemetry-2.14.0.jar:/usr/share/opensearch/lib/log4j-core-2.21.0.jar:/usr/share/opensearch/lib/joda-time-2.12.7.jar:/usr/share/opensearch/lib/lucene-misc-9.10.0.jar:/usr/share/opensearch/lib/log4j-api-2.21.0.jar:/usr/share/opensearch/lib/jackson-dataformat-cbor-2.17.0.jar:/usr/share/opensearch/lib/opensearch-common-2.14.0.jar:/usr/share/opensearch/lib/jackson-dataformat-smile-2.17.0.jar:/usr/share/opensearch/lib/t-digest-3.2.jar:/usr/share/opensearch/lib/opensearch-x-content-2.14.0.jar:/usr/share/opensearch/lib/log4j-jul-2.21.0.jar:/usr/share/opensearch/lib/lucene-sandbox-9.10.0.jar:/usr/share/opensearch/lib/lucene-backward-codecs-9.10.0.jar:/usr/share/opensearch/lib/snakeyaml-2.1.jar:/usr/share/opensearch/lib/opensearch-geo-2.14.0.jar:/usr/share/opensearch/lib/opensearch-2.13.0.jar:/usr/share/opensearch/lib/lucene-suggest-9.10.0.jar
[2024-10-02T13:17:31,681][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] sun.boot.class.path: null
[2024-10-02T13:17:31,684][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] java.home: /usr/share/opensearch/jdk
[2024-10-02T13:17:31,684][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar
[2024-10-02T13:17:31,686][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/lucene-spatial-extras-9.10.0.jar
[2024-10-02T13:17:31,687][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/lucene-memory-9.10.0.jar
[2024-10-02T13:17:31,687][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-cli-2.13.0.jar
[2024-10-02T13:17:31,688][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
[2024-10-02T13:17:31,688][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/java-version-checker-2.14.0.jar
[2024-10-02T13:17:31,689][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-compress-2.13.0.jar
[2024-10-02T13:17:31,689][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-core-2.14.0.jar
[2024-10-02T13:17:31,690][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
[2024-10-02T13:17:31,694][ERROR][o.o.b.Bootstrap          ] [a4212-os-master-2] Exception
java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.14.0.jar:2.14.0]
[2024-10-02T13:17:31,699][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [a4212-os-master-2] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.14.0.jar:2.14.0]
Caused by: java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.14.0.jar:2.14.0]
        ... 6 more
[2024-10-02T13:18:18,905][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Looking in classpath from jdk.internal.loader.ClassLoaders$AppClassLoader@4f2410ac for /com/sun/jna/linux-x86-64/libjnidispatch.so
[2024-10-02T13:18:18,908][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Found library resource at jar:file:/usr/share/opensearch/lib/jna-5.13.0.jar!/com/sun/jna/linux-x86-64/libjnidispatch.so
[2024-10-02T13:18:18,910][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Extracting library to /usr/share/opensearch/.cache/JNA/temp/jna16960683428097951911.tmp
[2024-10-02T13:18:18,912][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Trying /usr/share/opensearch/.cache/JNA/temp/jna16960683428097951911.tmp
[2024-10-02T13:18:18,914][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Found jnidispatch at /usr/share/opensearch/.cache/JNA/temp/jna16960683428097951911.tmp
[2024-10-02T13:18:18,959][DEBUG][o.o.b.SystemCallFilter   ] [a4212-os-master-2] Linux seccomp filter installation successful, threads: [all]
[2024-10-02T13:18:19,373][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] java.class.path: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar:/usr/share/opensearch/lib/lucene-spatial-extras-9.10.0.jar:/usr/share/opensearch/lib/lucene-memory-9.10.0.jar:/usr/share/opensearch/lib/opensearch-cli-2.13.0.jar:/usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar:/usr/share/opensearch/lib/java-version-checker-2.14.0.jar:/usr/share/opensearch/lib/opensearch-compress-2.13.0.jar:/usr/share/opensearch/lib/opensearch-core-2.14.0.jar:/usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar:/usr/share/opensearch/lib/java-version-checker-2.13.0.jar:/usr/share/opensearch/lib/opensearch-2.14.0.jar:/usr/share/opensearch/lib/jna-5.13.0.jar:/usr/share/opensearch/lib/opensearch-core-2.13.0.jar:/usr/share/opensearch/lib/lucene-grouping-9.10.0.jar:/usr/share/opensearch/lib/opensearch-compress-2.14.0.jar:/usr/share/opensearch/lib/lucene-spatial3d-9.10.0.jar:/usr/share/opensearch/lib/lucene-analysis-common-9.10.0.jar:/usr/share/opensearch/lib/joda-time-2.12.2.jar:/usr/share/opensearch/lib/opensearch-launchers-2.13.0.jar:/usr/share/opensearch/lib/opensearch-secure-sm-2.14.0.jar:/usr/share/opensearch/lib/jakarta.annotation-api-1.3.5.jar:/usr/share/opensearch/lib/protobuf-java-3.22.3.jar:/usr/share/opensearch/lib/jts-core-1.15.0.jar:/usr/share/opensearch/lib/opensearch-cli-2.14.0.jar:/usr/share/opensearch/lib/jackson-dataformat-yaml-2.17.0.jar:/usr/share/opensearch/lib/lucene-join-9.10.0.jar:/usr/share/opensearch/lib/reactive-streams-1.0.4.jar:/usr/share/opensearch/lib/zstd-jni-1.5.5-5.jar:/usr/share/opensearch/lib/opensearch-geo-2.13.0.jar:/usr/share/opensearch/lib/spatial4j-0.7.jar:/usr/share/opensearch/lib/opensearch-common-2.13.0.jar:/usr/share/opensearch/lib/opensearch-launchers-2.14.0.jar:/usr/share/opensearch/lib/opensearch-x-content-2.13.0.jar:/usr/share/opensearch/lib/reactor-core-3.5.15.jar:/usr/share/opensearch/lib/jackson-core-2.17.0.jar:/usr/share/opensearch/lib/lucene-core-9.10.0.jar:/usr/share/opensearch/lib/lucene-queryparser-9.10.0.jar:/usr/share/opensearch/lib/opensearch-telemetry-2.13.0.jar:/usr/share/opensearch/lib/jzlib-1.1.3.jar:/usr/share/opensearch/lib/lucene-highlighter-9.10.0.jar:/usr/share/opensearch/lib/opensearch-secure-sm-2.13.0.jar:/usr/share/opensearch/lib/jopt-simple-5.0.4.jar:/usr/share/opensearch/lib/HdrHistogram-2.1.12.jar:/usr/share/opensearch/lib/opensearch-telemetry-2.14.0.jar:/usr/share/opensearch/lib/log4j-core-2.21.0.jar:/usr/share/opensearch/lib/joda-time-2.12.7.jar:/usr/share/opensearch/lib/lucene-misc-9.10.0.jar:/usr/share/opensearch/lib/log4j-api-2.21.0.jar:/usr/share/opensearch/lib/jackson-dataformat-cbor-2.17.0.jar:/usr/share/opensearch/lib/opensearch-common-2.14.0.jar:/usr/share/opensearch/lib/jackson-dataformat-smile-2.17.0.jar:/usr/share/opensearch/lib/t-digest-3.2.jar:/usr/share/opensearch/lib/opensearch-x-content-2.14.0.jar:/usr/share/opensearch/lib/log4j-jul-2.21.0.jar:/usr/share/opensearch/lib/lucene-sandbox-9.10.0.jar:/usr/share/opensearch/lib/lucene-backward-codecs-9.10.0.jar:/usr/share/opensearch/lib/snakeyaml-2.1.jar:/usr/share/opensearch/lib/opensearch-geo-2.14.0.jar:/usr/share/opensearch/lib/opensearch-2.13.0.jar:/usr/share/opensearch/lib/lucene-suggest-9.10.0.jar
[2024-10-02T13:18:19,374][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] sun.boot.class.path: null
[2024-10-02T13:18:19,377][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] java.home: /usr/share/opensearch/jdk
[2024-10-02T13:18:19,378][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar
[2024-10-02T13:18:19,380][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/lucene-spatial-extras-9.10.0.jar
[2024-10-02T13:18:19,381][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/lucene-memory-9.10.0.jar
[2024-10-02T13:18:19,381][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-cli-2.13.0.jar
[2024-10-02T13:18:19,382][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
[2024-10-02T13:18:19,382][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/java-version-checker-2.14.0.jar
[2024-10-02T13:18:19,383][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-compress-2.13.0.jar
[2024-10-02T13:18:19,383][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-core-2.14.0.jar
[2024-10-02T13:18:19,384][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
[2024-10-02T13:18:19,390][ERROR][o.o.b.Bootstrap          ] [a4212-os-master-2] Exception
java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.14.0.jar:2.14.0]
[2024-10-02T13:18:19,395][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [a4212-os-master-2] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.14.0.jar:2.14.0]
Caused by: java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.14.0.jar:2.14.0]
        ... 6 more
~

[gaiksaya]

Adding @saravanan30erd to take a look.

[saravanan30erd]

@gaiksaya @dzaczek

Usually jar hell happens due to jar conflicts on same lib and removing the old lib can solve these issues sometimes. But what I didn't understand is why its not the problem until now and after 2.13 we are getting this issues.

@dzaczek Can you please give it a try by removing the old lib version and start the service?

jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar

[dzaczek]

@saravanan30erd after remove jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar

java.lang.IllegalStateException: jar hell!                                                                                                     │
class: org.apache.lucene.queries.CommonTermsQuery                                                                                              │
jar1: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar                                                                                      │
jar2: /usr/share/opensearch/lib/lucene-queries-9.11.1.jar                                                                                      │
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.17.1]                                │
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.17.1]                              │
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.17.1]                              │
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.17.1.jar:2.17.1]                                        │
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.17.1.jar:2.17.1]                                          │
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.17.1.jar:2.17.1]                                        │
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.17.1.jar:2.17.1]                                     │
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.17.1.jar:2.17.1]                 │
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.13.0.jar:2.17.1]                            │
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.13.0.jar:2.17.1]                                                │
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.17.1.jar:2.17.1]                                        │
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.17.1.jar:2.17.1]                                        │
[2024-10-15T09:39:43,247][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [a4211-os-master-1] uncaught exception in thread [main]             │
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!                                                          │
class: org.apache.lucene.queries.CommonTermsQuery                                                                                              │
jar1: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar                                                                                      │
jar2: /usr/share/opensearch/lib/lucene-queries-9.11.1.jar                                                                                      │
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.17.1.jar:2.17.1]                                       │
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.17.1.jar:2.17.1]                                    │
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.17.1.jar:2.17.1]                │
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.13.0.jar:2.17.1]                           │
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.13.0.jar:2.17.1]                                               │
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.17.1.jar:2.17.1]                                       │
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.17.1.jar:2.17.1]                                       │
Caused by: java.lang.IllegalStateException: jar hell!                                                                                          │
class: org.apache.lucene.queries.CommonTermsQuery                                                                                              │
jar1: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar                                                                                      │
jar2: /usr/share/opensearch/lib/lucene-queries-9.11.1.jar                                                                                      │
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.17.1]                                │
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.17.1]                              │
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.17.1]                              │
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.17.1.jar:2.17.1]                                        │
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.17.1.jar:2.17.1]                                         │
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.17.1.jar:2.17.1]                                       │
        ... 6 more                                                                                                                             │
                                                                                                                                               │

[saravanan30erd]

@dzaczek looks like conflict continues on other libs. Could you please try to remove all the files in this folder /usr/share/opensearch/lib and do the upgrade? If it works, we can add that step.

[dzaczek]

I addded top playbbok remove ../lib dir and after tahat problem with plugins

[2024-10-16T11:20:37,284][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:20:37,381][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:20:53,198][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:20:53,262][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:21:00,682][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:21:00,768][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:21:14,823][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:21:14,875][INFO ][o.o.p.PluginsService     ] [a4211-os-master-1] PluginService:onIndexModule index:[call_and_sms2024.05.04/t8Zu1AqXRzSBb_qq2i3A9w]
[2024-10-16T11:21:14,897][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:21:15,442][INFO ][o.o.c.c.Coordinator      ] [a4211-os-master-1] cluster-manager node [{a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true}] failed, restarting discovery
org.opensearch.transport.NodeDisconnectedException: [a4213-os-master-3][XXX.XXX.XXX.213:9300][disconnected] disconnected
[2024-10-16T11:21:15,460][INFO ][o.o.c.s.ClusterApplierService] [a4211-os-master-1] cluster-manager node changed {previous [{a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true}], current []}, term: 39, version: 97174, reason: becoming candidate: onLeaderFailure
[2024-10-16T11:21:15,515][WARN ][o.o.c.NodeConnectionsService] [a4211-os-master-1] failed to connect to {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true} (tried [1] times)
org.opensearch.transport.ConnectTransportException: [a4213-os-master-3][XXX.XXX.XXX.213:9300] connect_exception
        at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1077) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:217) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:57) ~[opensearch-common-2.13.0.jar:2.13.0]
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) ~[?:?]
        at org.opensearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:72) ~[opensearch-common-2.13.0.jar:2.13.0]
        at org.opensearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:83) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:322) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:338) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: XXX.XXX.XXX.213/XXX.XXX.XXX.213:9300
Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682) ~[?:?]
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973) ~[?:?]
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:335) ~[?:?]
        ... 7 more
[2024-10-16T11:21:15,428][WARN ][o.o.i.c.IndicesClusterStateService] [a4211-os-master-1] [call_and_sms2024.05.29][3] marking and sending shard failed due to [failed recovery]
org.opensearch.indices.recovery.RecoveryFailedException: [call_and_sms2024.05.29][3]: Recovery failed from {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true} into {a4211-os-master-1}{-pONGn5gRba3FaG7NlqnjQ}{v0q3HL2fQaOGwhw8jpQFsg}{XXX.XXX.XXX.211}{XXX.XXX.XXX.211:9300}{dm}{shard_indexing_pressure_enabled=true} ([call_and_sms2024.05.29][3]: Recovery failed from {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true} into {a4211-os-master-1}{-pONGn5gRba3FaG7NlqnjQ}{v0q3HL2fQaOGwhw8jpQFsg}{XXX.XXX.XXX.211}{XXX.XXX.XXX.211:9300}{dm}{shard_indexing_pressure_enabled=true})
        at org.opensearch.indices.recovery.RecoveryTarget.notifyListener(RecoveryTarget.java:136) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.replication.common.ReplicationTarget.fail(ReplicationTarget.java:180) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.replication.common.ReplicationCollection.fail(ReplicationCollection.java:212) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.PeerRecoveryTargetService$RecoveryResponseHandler.onException(PeerRecoveryTargetService.java:756) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.PeerRecoveryTargetService$RecoveryResponseHandler.handleException(PeerRecoveryTargetService.java:682) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:404) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1515) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.InboundHandler.lambda$handleException$5(InboundHandler.java:447) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) [opensearch-2.13.0.jar:2.13.0]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.opensearch.indices.recovery.RecoveryFailedException: [call_and_sms2024.05.29][3]: Recovery failed from {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true} into {a4211-os-master-1}{-pONGn5gRba3FaG7NlqnjQ}{v0q3HL2fQaOGwhw8jpQFsg}{XXX.XXX.XXX.211}{XXX.XXX.XXX.211:9300}{dm}{shard_indexing_pressure_enabled=true}
        ... 9 more
Caused by: org.opensearch.transport.RemoteTransportException: [a4213-os-master-3][XXX.XXX.XXX.213:9300][internal:index/shard/recovery/start_recovery]
Caused by: org.opensearch.transport.SendRequestTransportException: [a4211-os-master-1][XXX.XXX.XXX.211:9300][internal:index/shard/recovery/file_chunk]
        at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1002) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:267) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin$6$2.sendRequest(OpenSearchSecurityPlugin.java:847) ~[?:?]
        at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1750) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:885) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:862) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.RetryableTransportClient$1.tryAction(RetryableTransportClient.java:91) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.RetryableAction$1.doRun(RetryableAction.java:139) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:343) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.RetryableAction.run(RetryableAction.java:117) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.RetryableTransportClient.executeRetryableAction(RetryableTransportClient.java:106) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.replication.RemoteSegmentFileChunkWriter.writeFileChunk(RemoteSegmentFileChunkWriter.java:117) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.RemoteRecoveryTargetHandler.writeFileChunk(RemoteRecoveryTargetHandler.java:279) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.replication.SegmentFileTransferHandler$1.executeChunkRequest(SegmentFileTransferHandler.java:148) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.replication.SegmentFileTransferHandler$1.executeChunkRequest(SegmentFileTransferHandler.java:97) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.MultiChunkTransfer.handleItems(MultiChunkTransfer.java:149) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.MultiChunkTransfer$1.write(MultiChunkTransfer.java:98) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AsyncIOProcessor.processList(AsyncIOProcessor.java:131) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AsyncIOProcessor.drainAndProcessAndRelease(AsyncIOProcessor.java:119) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AsyncIOProcessor.put(AsyncIOProcessor.java:97) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.MultiChunkTransfer.addItem(MultiChunkTransfer.java:109) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.MultiChunkTransfer.lambda$handleItems$3(MultiChunkTransfer.java:151) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$4.onResponse(ActionListener.java:182) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.RetryableAction$RetryingListener.onResponse(RetryableAction.java:183) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:70) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:398) ~[?:?]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1505) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:420) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.InboundHandler.lambda$handleResponse$3(InboundHandler.java:414) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) ~[opensearch-2.13.0.jar:2.13.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.opensearch.node.NodeClosedException: node closed {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:983) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:267) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin$6$2.sendRequest(OpenSearchSecurityPlugin.java:847) ~[?:?]
        at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1750) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:885) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:862) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.RetryableTransportClient$1.tryAction(RetryableTransportClient.java:91) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.RetryableAction$1.doRun(RetryableAction.java:139) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:343) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.RetryableAction.run(RetryableAction.java:117) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.RetryableTransportClient.executeRetryableAction(RetryableTransportClient.java:106) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.replication.RemoteSegmentFileChunkWriter.writeFileChunk(RemoteSegmentFileChunkWriter.java:117) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.RemoteRecoveryTargetHandler.writeFileChunk(RemoteRecoveryTargetHandler.java:279) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.replication.SegmentFileTransferHandler$1.executeChunkRequest(SegmentFileTransferHandler.java:148) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.replication.SegmentFileTransferHandler$1.executeChunkRequest(SegmentFileTransferHandler.java:97) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.MultiChunkTransfer.handleItems(MultiChunkTransfer.java:149) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.MultiChunkTransfer$1.write(MultiChunkTransfer.java:98) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AsyncIOProcessor.processList(AsyncIOProcessor.java:131) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AsyncIOProcessor.drainAndProcessAndRelease(AsyncIOProcessor.java:119) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AsyncIOProcessor.put(AsyncIOProcessor.java:97) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.MultiChunkTransfer.addItem(MultiChunkTransfer.java:109) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.MultiChunkTransfer.lambda$handleItems$3(MultiChunkTransfer.java:151) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$4.onResponse(ActionListener.java:182) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.RetryableAction$RetryingListener.onResponse(RetryableAction.java:183) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:70) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:398) ~[?:?]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1505) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:420) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.InboundHandler.lambda$handleResponse$3(InboundHandler.java:414) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) ~[opensearch-2.13.0.jar:2.13.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-10-16T11:21:15,551][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:21:15,643][INFO ][o.o.n.Node               ] [a4211-os-master-1] stopping ...
[2024-10-16T11:21:15,656][WARN ][o.o.c.a.s.ShardStateAction] [a4211-os-master-1] node closed while execution action [internal:cluster/shard/failure] for shard entry [shard id [[call_and_sms2024.05.29][3]], allocation id [OH1e-x34SiqLvDxVjQOfUw], primary term [0], message [failed recovery], failure [RecoveryFailedException[[call_and_sms2024.05.29][3]: Recovery failed from {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true} into {a4211-os-master-1}{-pONGn5gRba3FaG7NlqnjQ}{v0q3HL2fQaOGwhw8jpQFsg}{XXX.XXX.XXX.211}{XXX.XXX.XXX.211:9300}{dm}{shard_indexing_pressure_enabled=true} ([call_and_sms2024.05.29][3]: Recovery failed from {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true} into {a4211-os-master-1}{-pONGn5gRba3FaG7NlqnjQ}{v0q3HL2fQaOGwhw8jpQFsg}{XXX.XXX.XXX.211}{XXX.XXX.XXX.211:9300}{dm}{shard_indexing_pressure_enabled=true})]; nested: RecoveryFailedException[[call_and_sms2024.05.29][3]: Recovery failed from {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true} into {a4211-os-master-1}{-pONGn5gRba3FaG7NlqnjQ}{v0q3HL2fQaOGwhw8jpQFsg}{XXX.XXX.XXX.211}{XXX.XXX.XXX.211:9300}{dm}{shard_indexing_pressure_enabled=true}]; nested: RemoteTransportException[[a4213-os-master-3][XXX.XXX.XXX.213:9300][internal:index/shard/recovery/start_recovery]]; nested: SendRequestTransportException[[a4211-os-master-1][XXX.XXX.XXX.211:9300][internal:index/shard/recovery/file_chunk]]; nested: NodeClosedException[node closed {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true}]; ], markAsStale [true]]
[2024-10-16T11:21:15,658][INFO ][o.o.i.r.PeerRecoveryTargetService] [a4211-os-master-1] recovery of [call_and_sms2024.05.04][4] from [{a4216-os-data-6}{RLKK7wQVRlaSWq3ETW0MpA}{i-eMpNxuRySu9fUJbxnT8w}{XXX.XXX.XXX.216}{XXX.XXX.XXX.216:9300}{d}{shard_indexing_pressure_enabled=true}] interrupted by network disconnect, will retry in [5s]; cause: [[a4216-os-data-6][XXX.XXX.XXX.216:9300][internal:index/shard/recovery/start_recovery] disconnected]
[2024-10-16T11:21:15,673][WARN ][o.o.d.HandshakingTransportAddressConnector] [a4211-os-master-1] handshake failed for [connectToRemoteMasterNode[XXX.XXX.XXX.218:9300]]
org.opensearch.transport.SendRequestTransportException: [][XXX.XXX.XXX.218:9300][internal:transport/handshake]
        at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1002) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:267) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.security.OpenSearchSecurityPlugin$6$2.sendRequest(OpenSearchSecurityPlugin.java:847) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1750) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:885) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.handshake(TransportService.java:668) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.handshake(TransportService.java:640) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.discovery.HandshakingTransportAddressConnector$1$1.onResponse(HandshakingTransportAddressConnector.java:133) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.discovery.HandshakingTransportAddressConnector$1$1.onResponse(HandshakingTransportAddressConnector.java:126) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$4.onResponse(ActionListener.java:182) [opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.ThreadedActionListener$1.doRun(ThreadedActionListener.java:78) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.13.0.jar:2.13.0]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.opensearch.node.NodeClosedException: node closed {a4211-os-master-1}{-pONGn5gRba3FaG7NlqnjQ}{v0q3HL2fQaOGwhw8jpQFsg}{XXX.XXX.XXX.211}{XXX.XXX.XXX.211:9300}{dm}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:983) ~[opensearch-2.13.0.jar:2.13.0]
        ... 15 more
[2024-10-16T11:21:15,674][WARN ][o.o.d.HandshakingTransportAddressConnector] [a4211-os-master-1] handshake failed for [connectToRemoteMasterNode[XXX.XXX.XXX.216:9300]]
org.opensearch.transport.SendRequestTransportException: [][XXX.XXX.XXX.216:9300][internal:transport/handshake]
        at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1002) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:267) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.security.OpenSearchSecurityPlugin$6$2.sendRequest(OpenSearchSecurityPlugin.java:847) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1750) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:885) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.handshake(TransportService.java:668) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.handshake(TransportService.java:640) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.discovery.HandshakingTransportAddressConnector$1$1.onResponse(HandshakingTransportAddressConnector.java:133) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.discovery.HandshakingTransportAddressConnector$1$1.onResponse(HandshakingTransportAddressConnector.java:126) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$4.onResponse(ActionListener.java:182) [opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.ThreadedActionListener$1.doRun(ThreadedActionListener.java:78) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.13.0.jar:2.13.0]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.opensearch.node.NodeClosedException: node closed {a4211-os-master-1}{-pONGn5gRba3FaG7NlqnjQ}{v0q3HL2fQaOGwhw8jpQFsg}{XXX.XXX.XXX.211}{XXX.XXX.XXX.211:9300}{dm}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:983) ~[opensearch-2.13.0.jar:2.13.0]
        ... 15 more
[2024-10-16T11:21:15,677][INFO ][o.o.s.a.r.AuditMessageRouter] [a4211-os-master-1] Closing AuditMessageRouter
[2024-10-16T11:21:15,678][INFO ][o.o.s.a.s.SinkProvider   ] [a4211-os-master-1] Closing InternalOpenSearchSink
[2024-10-16T11:21:15,678][INFO ][o.o.s.a.s.SinkProvider   ] [a4211-os-master-1] Closing DebugSink
[2024-10-16T11:21:18,085][INFO ][o.o.n.Node               ] [a4211-os-master-1] stopped
[2024-10-16T11:21:18,085][INFO ][o.o.n.Node               ] [a4211-os-master-1] closing ...
[2024-10-16T11:21:18,094][INFO ][o.o.s.a.i.AuditLogImpl   ] [a4211-os-master-1] Closing AuditLogImpl
[2024-10-16T11:21:18,107][INFO ][o.o.n.Node               ] [a4211-os-master-1] closed
[2024-10-16T11:21:25,397][ERROR][o.o.b.Bootstrap          ] [a4211-os-master-1] Exception
java.lang.IllegalStateException: codebase property already set: codebase.opensearch-geospatial -> file:/usr/share/opensearch/plugins/opensearch-geospatial/opensearch-geospatial-2.13.0.0.jar, cannot set to file:/usr/share/opensearch/plugins/opensearch-geospatial/opensearch-geospatial-2.17.1.0.jar
        at org.opensearch.bootstrap.Security.addCodebaseToSystemProperties(Security.java:297) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.readPolicy(Security.java:258) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.getPluginPermissions(Security.java:211) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.configure(Security.java:145) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:237) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.17.1.jar:2.17.1]
[2024-10-16T11:21:25,407][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [a4211-os-master-1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: codebase property already set: codebase.opensearch-geospatial -> file:/usr/share/opensearch/plugins/opensearch-geospatial/opensearch-geospatial-2.13.0.0.jar, cannot set to file:/usr/share/opensearch/plugins/opensearch-geospatial/opensearch-geospatial-2.17.1.0.jar
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.17.1.jar:2.17.1]
Caused by: java.lang.IllegalStateException: codebase property already set: codebase.opensearch-geospatial -> file:/usr/share/opensearch/plugins/opensearch-geospatial/opensearch-geospatial-2.13.0.0.jar, cannot set to file:/usr/share/opensearch/plugins/opensearch-geospatial/opensearch-geospatial-2.17.1.0.jar
        at org.opensearch.bootstrap.Security.addCodebaseToSystemProperties(Security.java:297) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.readPolicy(Security.java:258) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.getPluginPermissions(Security.java:211) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.configure(Security.java:145) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:237) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.17.1.jar:2.17.1]

[dzaczek]

then i tied remove plugins and lib

 org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.17.1.jar:2.17.1]
Caused by: java.lang.IllegalStateException: codebase property already set: codebase.jackson-databind -> file:/usr/share/opensearch/modules/ingest-geoip/jackson-databind-2.17.0.jar, cannot set to file:/usr/share/opensearch/modules/ingest-geoip/jackson-databind-2.17.2.jar
        at org.opensearch.bootstrap.Security.addCodebaseToSystemProperties(Security.java:297) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.readPolicy(Security.java:258) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.getPluginPermissions(Security.java:211) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.configure(Security.java:145) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:237) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.17.1.jar:2.17.1]
        ... 6 more
[2024-10-16T11:38:28,798][ERROR][o.o.b.Bootstrap          ] [a4211-os-master-1] Exception
java.lang.IllegalStateException: codebase property already set: codebase.jackson-databind -> file:/usr/share/opensearch/modules/ingest-geoip/jackson-databind-2.17.0.jar, cannot set to file:/usr/share/opensearch/modules/ingest-geoip/jackson-databind-2.17.2.jar
        at org.opensearch.bootstrap.Security.addCodebaseToSystemProperties(Security.java:297) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.readPolicy(Security.java:258) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.getPluginPermissions(Security.java:211) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.configure(Security.java:145) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:237) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.17.1.jar:2.17.1]
[2024-10-16T11:38:28,806][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [a4211-os-master-1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: codebase property already set: codebase.jackson-databind -> file:/usr/share/opensearch/modules/ingest-geoip/jackson-databind-2.17.0.jar, cannot set to file:/usr/share/opensearch/modules/ingest-geoip/jackson-databind-2.17.2.jar
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.17.1.jar:2.17.1]
Caused by: java.lang.IllegalStateException: codebase property already set: codebase.jackson-databind -> file:/usr/share/opensearch/modules/ingest-geoip/jackson-databind-2.17.0.jar, cannot set to file:/usr/share/opensearch/modules/ingest-geoip/jackson-databind-2.17.2.jar
        at org.opensearch.bootstrap.Security.addCodebaseToSystemProperties(Security.java:297) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.readPolicy(Security.java:258) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.getPluginPermissions(Security.java:211) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.configure(Security.java:145) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:237) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.17.1.jar:2.17.1]
        ... 6 more
root@a4211-os-master-1:~#

Ok so propably i will do update manually :)

[dzaczek]

I prepared a workaround script to handle each node manually and preserve the config and data directories.

cd /usr/share 
/usr/share/opensearch/bin/opensearch-plugin remove prometheus-exporter
wget https://artifacts.opensearch.org/releases/bundle/opensearch/2.17.1/opensearch-2.17.1-linux-x64.tar.gz
tar -xvzf opensearch-2.17.1-linux-x64.tar.gz
cp /usr/share/opensearch/config/*  /usr/share/opensearch-2.17.1/config/
systemctl stop opensearch 
sleep 5
find /usr/share/opensearch/ -type f -not -path "/usr/share/opensearch/data/*" -not -path "/usr/share/opensearch/config/*" -delete
chown -R opensearch:opensearch /usr/share/opensearch-2.17.1/
sed -i 's|^#path\.data:.*|path.data: /usr/share/opensearch/data|' /usr/share/opensearch-2.17.1/opensearch.yml
rsync -av --exclude='data' /usr/share/opensearch-2.17.1/ /usr/share/opensearch/
chown -R opensearch:opensearch /usr/share/opensearch
/usr/share/opensearch/bin/opensearch-plugin install  https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/2.17.1.0/prometheus-exporter-2.17.1.0.zip

And also converted this same script into an Ansible playbook.

---
- name: Update OpenSearch on all hosts serially
  hosts: os-cluster
  serial: 1  # Ensure only one host is processed at a time
  become: true
  tasks:
    - name: Install rsync
      apt:
        name: rsync
        state: present
        update_cache: yes

    - name: Remove Prometheus exporter OpenSearch plugin
      ansible.builtin.command: /usr/share/opensearch/bin/opensearch-plugin remove prometheus-exporter
      ignore_errors: yes

    - name: Download OpenSearch 2.17.1
      ansible.builtin.get_url:
        url: https://artifacts.opensearch.org/releases/bundle/opensearch/2.17.1/opensearch-2.17.1-linux-x64.tar.gz
        dest: /usr/share/opensearch-2.17.1-linux-x64.tar.gz

    - name: Extract OpenSearch tarball
      ansible.builtin.unarchive:
        src: /usr/share/opensearch-2.17.1-linux-x64.tar.gz
        dest: /usr/share/
        remote_src: yes

    - name: Copy existing OpenSearch config to the new version
      ansible.builtin.copy:
        src: /usr/share/opensearch/config/
        dest: /usr/share/opensearch-2.17.1/config/
        owner: opensearch
        group: opensearch
        mode: '0644'
        remote_src: yes

    - name: Stop OpenSearch service
      ansible.builtin.systemd:
        name: opensearch
        state: stopped

    - name: Wait for 5 seconds after stopping OpenSearch
      ansible.builtin.pause:
        seconds: 5

    - name: Delete files from the current OpenSearch folder, except for data and config directories
      ansible.builtin.shell: |
        find /usr/share/opensearch/ -type f -not -path "/usr/share/opensearch/data/*" -not -path "/usr/share/opensearch/config/*" -delete

    - name: Change ownership of the new OpenSearch installation
      ansible.builtin.command: chown -R opensearch:opensearch /usr/share/opensearch-2.17.1/

    - name: Modify opensearch.yml to adjust data path
      ansible.builtin.lineinfile:
        path: /usr/share/opensearch-2.17.1/config/opensearch.yml
        regexp: '^#path\.data:.*'
        line: 'path.data: /usr/share/opensearch/data'

    - name: Synchronize new OpenSearch files (excluding data) with the old directory
      ansible.builtin.command: rsync -av --exclude='data' /usr/share/opensearch-2.17.1/ /usr/share/opensearch/

    - name: Ensure proper ownership of the OpenSearch folder
      ansible.builtin.command: chown -R opensearch:opensearch /usr/share/opensearch

    - name: Set ownership and permissions for OpenSearch config directory
      ansible.builtin.file:
        path: /usr/share/opensearch/config
        owner: opensearch
        group: opensearch
        mode: '0700'

    - name: Install Prometheus exporter plugin
      ansible.builtin.command: /usr/share/opensearch/bin/opensearch-plugin install https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/2.17.1.0/prometheus-exporter-2.17.1.0.zip

    - name: Start OpenSearch service
      ansible.builtin.systemd:
        name: opensearch
        state: started

    - name: Wait for 30 seconds for OpenSearch to start
      ansible.builtin.pause:
        seconds: 30

    - name: Check if OpenSearch service is running
      ansible.builtin.command: systemctl is-active opensearch
      register: open_search_status
      ignore_errors: yes

    - name: Debug OpenSearch status
      debug:
        var: open_search_status

    - name: Fail and stop playbook if OpenSearch did not start
      ansible.builtin.meta: end_play
      when: open_search_status.stdout != 'active'

    - name: Wait for 5 minutes if OpenSearch is running
      ansible.builtin.pause:
        minutes: 5
      when: open_search_status.stdout == 'active'