search

opensearch-project / opensearch-k8s-operator

OpenSearch Kubernetes Operator
Apache License 2.0
380 stars 200 forks source link

Install opensearch 2.5 with operator fails in kubernetes #439

Closed rigl1 closed 1 year ago

rigl1 commented 1 year ago

Hi, iam try to install the opensearch demo example in kuberntes. Example cluster v2

The operator runs in version 2.2.1. I got the known error log Gatekeeper error 2023/02/18 12:17:06 http: TLS handshake error from 172.16.233.192:50278: EOF

Kubernetes version: 1.23.9

I got this logs from the cluster start:

-> Installing repository-s3
-> Downloading repository-s3 from opensearch
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.NetPermission setDefaultAuthenticator
* java.net.SocketPermission * connect,resolve
* java.util.PropertyPermission opensearch.allow_insecure_settings read,write
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
-> Installed repository-s3 with folder name repository-s3
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
OpenSearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on NAME="Amazon Linux"
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-2.5.0
Detected OpenSearch Security Version: 2.5.0.0
/usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
Enabling OpenSearch Security Plugin
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.5.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.5.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2023-02-18T12:23:53,701][INFO ][o.o.n.Node               ] [my-first-cluster-masters-0] version[2.5.0], pid[99], build[tar/b8a8b6c4d7fc7a7e32eb2cb68ecad8057a4636ad/2023-01-18T23:48:48.981786100Z], OS[Linux/5.16.13-200.fc35.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.5/17.0.5+8]
[2023-02-18T12:23:53,703][INFO ][o.o.n.Node               ] [my-first-cluster-masters-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
[2023-02-18T12:23:53,703][INFO ][o.o.n.Node               ] [my-first-cluster-masters-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-18111117880549826994, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xmx512M, -Xms512M, -Dopensearch.transport.cname_in_publish_address=true, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2023-02-18T12:23:56,597][WARN ][stderr                   ] [my-first-cluster-masters-0] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2023-02-18T12:23:56,605][WARN ][stderr                   ] [my-first-cluster-masters-0] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2023-02-18T12:23:56,605][WARN ][stderr                   ] [my-first-cluster-masters-0] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2023-02-18T12:23:56,667][INFO ][o.o.s.s.t.SSLConfig      ] [my-first-cluster-masters-0] SSL dual mode is disabled
[2023-02-18T12:23:56,668][INFO ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] OpenSearch Config path is /usr/share/opensearch/config
[2023-02-18T12:23:57,405][INFO ][o.o.s.s.DefaultSecurityKeyStore] [my-first-cluster-masters-0] JVM supports TLSv1.3
[2023-02-18T12:23:57,407][INFO ][o.o.s.s.DefaultSecurityKeyStore] [my-first-cluster-masters-0] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[2023-02-18T12:23:58,991][INFO ][o.o.s.s.DefaultSecurityKeyStore] [my-first-cluster-masters-0] TLS Transport Client Provider : JDK
[2023-02-18T12:23:58,991][INFO ][o.o.s.s.DefaultSecurityKeyStore] [my-first-cluster-masters-0] TLS Transport Server Provider : JDK
[2023-02-18T12:23:58,991][INFO ][o.o.s.s.DefaultSecurityKeyStore] [my-first-cluster-masters-0] TLS HTTP Provider             : JDK
[2023-02-18T12:23:58,991][INFO ][o.o.s.s.DefaultSecurityKeyStore] [my-first-cluster-masters-0] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2023-02-18T12:23:58,992][INFO ][o.o.s.s.DefaultSecurityKeyStore] [my-first-cluster-masters-0] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2]
[2023-02-18T12:23:59,014][INFO ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] Clustername: my-first-cluster
[2023-02-18T12:23:59,024][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
[2023-02-18T12:23:59,025][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,025][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] Directory /usr/share/opensearch/config/tls-transport has insecure file permissions (should be 0700)
[2023-02-18T12:23:59,025][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-transport/my-first-cluster-masters-0.key has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,026][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-transport/my-first-cluster-masters-0.crt has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,026][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-transport/my-first-cluster-bootstrap-0.key has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,026][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-transport/my-first-cluster-bootstrap-0.crt has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,026][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-transport/ca.crt has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,027][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-transport/..data has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,027][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] Directory /usr/share/opensearch/config/tls-transport/..2023_02_18_12_23_13.229486537 has insecure file permissions (should be 0700)
[2023-02-18T12:23:59,027][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-transport/..2023_02_18_12_23_13.229486537/my-first-cluster-masters-0.key has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,027][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-transport/..2023_02_18_12_23_13.229486537/my-first-cluster-masters-0.crt has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,028][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-transport/..2023_02_18_12_23_13.229486537/my-first-cluster-bootstrap-0.key has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,028][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-transport/..2023_02_18_12_23_13.229486537/my-first-cluster-bootstrap-0.crt has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,028][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-transport/..2023_02_18_12_23_13.229486537/ca.crt has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,028][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] Directory /usr/share/opensearch/config/tls-http has insecure file permissions (should be 0700)
[2023-02-18T12:23:59,029][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-http/tls.key has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,029][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-http/tls.crt has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,029][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-http/ca.crt has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,029][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-http/..data has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,029][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] Directory /usr/share/opensearch/config/tls-http/..2023_02_18_12_23_13.724882806 has insecure file permissions (should be 0700)
[2023-02-18T12:23:59,029][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-http/..2023_02_18_12_23_13.724882806/tls.crt has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,030][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-http/..2023_02_18_12_23_13.724882806/ca.crt has insecure file permissions (should be 0600)
[2023-02-18T12:23:59,030][WARN ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] File /usr/share/opensearch/config/tls-http/..2023_02_18_12_23_13.724882806/tls.key has insecure file permissions (should be 0600)
[2023-02-18T12:24:01,078][INFO ][o.o.p.c.PluginSettings   ] [my-first-cluster-masters-0] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2023-02-18T12:24:02,116][INFO ][o.o.i.r.ReindexPlugin    ] [my-first-cluster-masters-0] ReindexPlugin reloadSPI called
[2023-02-18T12:24:02,117][INFO ][o.o.i.r.ReindexPlugin    ] [my-first-cluster-masters-0] Unable to find any implementation for RemoteReindexExtension
[2023-02-18T12:24:02,261][INFO ][o.o.j.JobSchedulerPlugin ] [my-first-cluster-masters-0] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2023-02-18T12:24:02,288][INFO ][o.o.j.JobSchedulerPlugin ] [my-first-cluster-masters-0] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2023-02-18T12:24:02,358][INFO ][o.o.j.JobSchedulerPlugin ] [my-first-cluster-masters-0] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2023-02-18T12:24:02,385][INFO ][o.o.j.JobSchedulerPlugin ] [my-first-cluster-masters-0] Loaded scheduler extension: observability, index: .opensearch-observability-job
[2023-02-18T12:24:02,395][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [aggs-matrix-stats]
[2023-02-18T12:24:02,395][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [analysis-common]
[2023-02-18T12:24:02,395][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [geo]
[2023-02-18T12:24:02,395][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [ingest-common]
[2023-02-18T12:24:02,395][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [ingest-geoip]
[2023-02-18T12:24:02,395][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [ingest-user-agent]
[2023-02-18T12:24:02,396][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [lang-expression]
[2023-02-18T12:24:02,396][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [lang-mustache]
[2023-02-18T12:24:02,396][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [lang-painless]
[2023-02-18T12:24:02,396][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [mapper-extras]
[2023-02-18T12:24:02,396][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [opensearch-dashboards]
[2023-02-18T12:24:02,396][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [parent-join]
[2023-02-18T12:24:02,396][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [percolator]
[2023-02-18T12:24:02,396][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [rank-eval]
[2023-02-18T12:24:02,396][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [reindex]
[2023-02-18T12:24:02,396][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [repository-url]
[2023-02-18T12:24:02,396][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [systemd]
[2023-02-18T12:24:02,397][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded module [transport-netty4]
[2023-02-18T12:24:02,397][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-alerting]
[2023-02-18T12:24:02,397][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-anomaly-detection]
[2023-02-18T12:24:02,397][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-asynchronous-search]
[2023-02-18T12:24:02,397][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-cross-cluster-replication]
[2023-02-18T12:24:02,397][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-geospatial]
[2023-02-18T12:24:02,397][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-index-management]
[2023-02-18T12:24:02,397][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-job-scheduler]
[2023-02-18T12:24:02,398][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-knn]
[2023-02-18T12:24:02,398][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-ml]
[2023-02-18T12:24:02,398][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-neural-search]
[2023-02-18T12:24:02,398][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-notifications]
[2023-02-18T12:24:02,398][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-notifications-core]
[2023-02-18T12:24:02,398][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-observability]
[2023-02-18T12:24:02,398][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-performance-analyzer]
[2023-02-18T12:24:02,398][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-reports-scheduler]
[2023-02-18T12:24:02,398][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-security]
[2023-02-18T12:24:02,398][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-security-analytics]
[2023-02-18T12:24:02,398][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [opensearch-sql]
[2023-02-18T12:24:02,398][INFO ][o.o.p.PluginsService     ] [my-first-cluster-masters-0] loaded plugin [repository-s3]
[2023-02-18T12:24:02,497][INFO ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2023-02-18T12:24:02,574][INFO ][o.o.e.NodeEnvironment    ] [my-first-cluster-masters-0] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/vda4)]], net usable_space [39.8gb], net total_space [49.4gb], types [xfs]
[2023-02-18T12:24:02,574][INFO ][o.o.e.NodeEnvironment    ] [my-first-cluster-masters-0] heap size [512mb], compressed ordinary object pointers [true]
[2023-02-18T12:24:02,662][INFO ][o.o.n.Node               ] [my-first-cluster-masters-0] node name [my-first-cluster-masters-0], node ID [t3IFy5hSQaqEG9w1GV8zog], cluster name [my-first-cluster], roles [data, cluster_manager]
[2023-02-18T12:24:08,363][DEPRECATION][o.o.d.c.s.Settings       ] [my-first-cluster-masters-0] [cluster.initial_master_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2023-02-18T12:24:11,378][WARN ][o.o.s.c.Salt             ] [my-first-cluster-masters-0] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2023-02-18T12:24:11,470][INFO ][o.o.s.a.i.AuditLogImpl   ] [my-first-cluster-masters-0] Message routing enabled: true
[2023-02-18T12:24:11,568][INFO ][o.o.s.f.SecurityFilter   ] [my-first-cluster-masters-0] <NONE> indices are made immutable.
[2023-02-18T12:24:12,286][INFO ][o.o.a.b.ADCircuitBreakerService] [my-first-cluster-masters-0] Registered memory breaker.
[2023-02-18T12:24:13,057][INFO ][o.o.m.b.MLCircuitBreakerService] [my-first-cluster-masters-0] Registered ML memory breaker.
[2023-02-18T12:24:13,058][INFO ][o.o.m.b.MLCircuitBreakerService] [my-first-cluster-masters-0] Registered ML disk breaker.
[2023-02-18T12:24:13,059][INFO ][o.o.m.b.MLCircuitBreakerService] [my-first-cluster-masters-0] Registered ML native memory breaker.
[2023-02-18T12:24:13,267][INFO ][o.r.Reflections          ] [my-first-cluster-masters-0] Reflections took 91 ms to scan 1 urls, producing 12 keys and 32 values
[2023-02-18T12:24:15,786][INFO ][o.o.t.NettyAllocator     ] [my-first-cluster-masters-0] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
[2023-02-18T12:24:15,978][INFO ][o.o.d.DiscoveryModule    ] [my-first-cluster-masters-0] using discovery type [zen] and seed hosts providers [settings]
[2023-02-18T12:24:17,002][WARN ][o.o.g.DanglingIndicesState] [my-first-cluster-masters-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-02-18T12:24:18,099][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [my-first-cluster-masters-0] PerformanceAnalyzer Enabled: false
[2023-02-18T12:24:18,194][INFO ][o.o.n.Node               ] [my-first-cluster-masters-0] initialized
[2023-02-18T12:24:18,195][INFO ][o.o.n.Node               ] [my-first-cluster-masters-0] starting ...
[2023-02-18T12:24:18,500][INFO ][o.o.t.TransportService   ] [my-first-cluster-masters-0] publish_address {my-first-cluster-masters-0/10.100.39.78:9300}, bound_addresses {[::]:9300}
[2023-02-18T12:24:18,792][INFO ][o.o.b.BootstrapChecks    ] [my-first-cluster-masters-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-02-18T12:24:18,805][INFO ][o.o.c.c.ClusterBootstrapService] [my-first-cluster-masters-0] skipping cluster bootstrapping as local node does not match bootstrap requirements: [my-first-cluster-bootstrap-0]
[2023-02-18T12:24:28,808][WARN ][o.o.c.c.ClusterFormationFailureHelper] [my-first-cluster-masters-0] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [my-first-cluster-bootstrap-0] to bootstrap a cluster: have discovered [{my-first-cluster-masters-0}{t3IFy5hSQaqEG9w1GV8zog}{LHQ77zu1SJ2E2mMv3Rr16A}{my-first-cluster-masters-0}{10.100.39.78:9300}{dm}{shard_indexing_pressure_enabled=true}]; discovery will continue using [] from hosts providers and [{my-first-cluster-masters-0}{t3IFy5hSQaqEG9w1GV8zog}{LHQ77zu1SJ2E2mMv3Rr16A}{my-first-cluster-masters-0}{10.100.39.78:9300}{dm}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-02-18T12:24:38,810][WARN ][o.o.c.c.ClusterFormationFailureHelper] [my-first-cluster-masters-0] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [my-first-cluster-bootstrap-0] to bootstrap a cluster: have discovered [{my-first-cluster-masters-0}{t3IFy5hSQaqEG9w1GV8zog}{LHQ77zu1SJ2E2mMv3Rr16A}{my-first-cluster-masters-0}{10.100.39.78:9300}{dm}{shard_indexing_pressure_enabled=true}]; discovery will continue using [] from hosts providers and [{my-first-cluster-masters-0}{t3IFy5hSQaqEG9w1GV8zog}{LHQ77zu1SJ2E2mMv3Rr16A}{my-first-cluster-masters-0}{10.100.39.78:9300}{dm}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-02-18T12:24:48,810][WARN ][o.o.n.Node               ] [my-first-cluster-masters-0] timed out while waiting for initial discovery state - timeout: 30s
[2023-02-18T12:24:48,811][WARN ][o.o.c.c.ClusterFormationFailureHelper] [my-first-cluster-masters-0] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [my-first-cluster-bootstrap-0] to bootstrap a cluster: have discovered [{my-first-cluster-masters-0}{t3IFy5hSQaqEG9w1GV8zog}{LHQ77zu1SJ2E2mMv3Rr16A}{my-first-cluster-masters-0}{10.100.39.78:9300}{dm}{shard_indexing_pressure_enabled=true}]; discovery will continue using [] from hosts providers and [{my-first-cluster-masters-0}{t3IFy5hSQaqEG9w1GV8zog}{LHQ77zu1SJ2E2mMv3Rr16A}{my-first-cluster-masters-0}{10.100.39.78:9300}{dm}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-02-18T12:24:48,819][INFO ][o.o.h.AbstractHttpServerTransport] [my-first-cluster-masters-0] publish_address {my-first-cluster-masters-0/10.100.39.78:9200}, bound_addresses {[::]:9200}
[2023-02-18T12:24:48,819][INFO ][o.o.n.Node               ] [my-first-cluster-masters-0] started
[2023-02-18T12:24:48,820][INFO ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] Node started
[2023-02-18T12:24:48,820][INFO ][o.o.s.c.ConfigurationRepository] [my-first-cluster-masters-0] Will not attempt to create index .opendistro_security and default configs if they are absent. Use securityadmin to initialize cluster
[2023-02-18T12:24:48,821][INFO ][o.o.s.c.ConfigurationRepository] [my-first-cluster-masters-0] Background init thread started. Install default config?: false
[2023-02-18T12:24:48,821][INFO ][o.o.s.OpenSearchSecurityPlugin] [my-first-cluster-masters-0] 0 OpenSearch Security modules loaded so far: []
[2023-02-18T12:24:48,827][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [my-first-cluster-masters-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
    at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:205) ~[opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:191) ~[opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:461) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:577) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:208) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:99) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-02-18T12:24:53,600][ERROR][o.o.s.a.BackendRegistry  ] [my-first-cluster-masters-0] Not yet initialized (you may need to run securityadmin)
[2023-02-18T12:24:58,813][WARN ][o.o.c.c.ClusterFormationFailureHelper] [my-first-cluster-masters-0] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [my-first-cluster-bootstrap-0] to bootstrap a cluster: have discovered [{my-first-cluster-masters-0}{t3IFy5hSQaqEG9w1GV8zog}{LHQ77zu1SJ2E2mMv3Rr16A}{my-first-cluster-masters-0}{10.100.39.78:9300}{dm}{shard_indexing_pressure_enabled=true}]; discovery will continue using [] from hosts providers and [{my-first-cluster-masters-0}{t3IFy5hSQaqEG9w1GV8zog}{LHQ77zu1SJ2E2mMv3Rr16A}{my-first-cluster-masters-0}{10.100.39.78:9300}{dm}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-02-18T12:25:01,836][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [my-first-cluster-masters-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
    at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:205) ~[opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:191) ~[opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:461) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:577) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:208) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:99) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-02-18T12:25:08,814][WARN ][o.o.c.c.ClusterFormationFailureHelper] [my-first-cluster-masters-0] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [my-first-cluster-bootstrap-0] to bootstrap a cluster: have discovered [{my-first-cluster-masters-0}{t3IFy5hSQaqEG9w1GV8zog}{LHQ77zu1SJ2E2mMv3Rr16A}{my-first-cluster-masters-0}{10.100.39.78:9300}{dm}{shard_indexing_pressure_enabled=true}]; discovery will continue using [] from hosts providers and [{my-first-cluster-masters-0}{t3IFy5hSQaqEG9w1GV8zog}{LHQ77zu1SJ2E2mMv3Rr16A}{my-first-cluster-masters-0}{10.100.39.78:9300}{dm}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-02-18T12:25:13,383][ERROR][o.o.s.a.BackendRegistry  ] [my-first-cluster-masters-0] Not yet initialized (you may need to run securityadmin)
[2023-02-18T12:25:14,842][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [my-first-cluster-masters-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
    at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:205) ~[opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:191) ~[opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:461) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:577) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:208) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:99) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-02-18T12:25:18,816][WARN ][o.o.c.c.ClusterFormationFailureHelper] [my-first-cluster-masters-0] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [my-first-cluster-bootstrap-0] to bootstrap a cluster: have discovered [{my-first-cluster-masters-0}{t3IFy5hSQaqEG9w1GV8zog}{LHQ77zu1SJ2E2mMv3Rr16A}{my-first-cluster-masters-0}{10.100.39.78:9300}{dm}{shard_indexing_pressure_enabled=true}]; discovery will continue using [] from hosts providers and [{my-first-cluster-masters-0}{t3IFy5hSQaqEG9w1GV8zog}{LHQ77zu1SJ2E2mMv3Rr16A}{my-first-cluster-masters-0}{10.100.39.78:9300}{dm}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-02-18T12:25:27,844][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [my-first-cluster-masters-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
    at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:205) ~[opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:191) ~[opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:461) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:577) [opensearch-2.5.0.jar:2.5.0]
    at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:208) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:99) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.5.0.0.jar:2.5.0.0]
    at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-02-18T12:25:28,818][WARN ][o.o.c.c.ClusterFormationFailureHelper] [my-first-cluster-masters-0] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [my-first-cluster-bootstrap-0] to bootstrap a cluster: have discovered [{my-first-cluster-masters-0}{t3IFy5hSQaqEG9w1GV8zog}{LHQ77zu1SJ2E2mMv3Rr16A}{my-first-cluster-masters-0}{10.100.39.78:9300}{dm}{shard_indexing_pressure_enabled=true}]; discovery will continue using [] from hosts providers and [{my-first-cluster-masters-0}{t3IFy5hSQaqEG9w1GV8zog}{LHQ77zu1SJ2E2mMv3Rr16A}{my-first-cluster-masters-0}{10.100.39.78:9300}{dm}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
swoehrl-mw commented 1 year ago

Hi @rigl1. Without more details on your pods it's hard to diagnose, but my first guess would be that the bootstrap pod is not coming up. Please check the pod status. In the beginning there should be master-0 and bootstrap pods.

rigl1 commented 1 year ago

Hi! thank you for response. Should the services have a cluster ip ? image

rigl1 commented 1 year ago

We analyzed a little more. We use argocd to roll out the cluster in kubernetes. We believe that the annotation for the job starts too early and therefore the pod is not initialized correctly. We need to add the following variables to the helm chart:

swoehrl-mw commented 1 year ago

Hi @rigl1. I don't really understand in what way argocd is interfering here. Could you please go into more detail what is (not) happening and in what way the operator needs to do something?

rigl1 commented 1 year ago

Ticket can be closed

leonsteinhaeuser commented 1 year ago

@rigl1 can you please provide the solution that helped you to solve this problem? We are also facing this problem but we have not installed the cluster through argocd.

rigl1 commented 1 year ago

HI @leonsteinhaeuser unfortunately, I could not solve the problem with ArgoCD either. But what i know ... You have to use this variables in Argo yaml to force Argo to wait.

Additional annotations for jobs pods additionalJobAnnotations: {}

example ArgoCD requires its specific annotations "argocd.argoproj.io/hook": Sync "argocd.argoproj.io/sync-wave": "0" "argocd.argoproj.io/hook-delete-policy": BeforeHookCreation,HookSucceeded

Youpiiiii commented 3 weeks ago

Hi @rigl1,

I don't understand. Where do you put the argocd annotations? I can't find additionalJobAnnotations.

Thank you