search

wazuh / wazuh-kubernetes

Wazuh - Wazuh Kubernetes
https://wazuh.com/
GNU General Public License v2.0
263 stars 163 forks source link

Indexer crash loop without providing error message #879

Open karlschriek opened 18 hours ago

karlschriek commented 18 hours ago

I am deploying Wazuh based on the kubernetes manifests in this repo. I've made adjustments in order to generate secure credentials instead of using the hardcoded secrets, and also to generate TLS certs using certificate manager. But other than that the manifests are pretty much the same as the example ones provided.

I am deploying v4.9.1.

Dashboard and manager (master plus workers) start up correctly. But the indexer is in a crash loop. It starts up, runs for about 60 seconds and then restarts. It does not provide any error messages that would indicate the reason for the crash, which makes it extremely hard to debug.

There are a lot of warnings, but since those are unlikely to have anything to do with changes I've made, I assume they should not play any kind of important role here.

Here are the complete logs for one of the pods. Does any of this point to an obvious problem? Or does anyone have suggestions on how I could discover the root problem?

WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
Oct 31, 2024 3:43:00 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2024-10-31T15:43:02,022][INFO ][o.o.n.Node               ] [wazuh-indexer-0] version[2.13.0], pid[1], build[rpm/06e21c13dd7df95b42014376ce7531fa574ce569/2024-10-15T16:48:17.780639Z], OS[Linux/5.15.0-1064-azure/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.2/21.0.2+13-LTS]
[2024-10-31T15:43:02,024][INFO ][o.o.n.Node               ] [wazuh-indexer-0] JVM home [/usr/share/wazuh-indexer/jdk], using bundled JDK/JRE [true]
[2024-10-31T15:43:02,024][INFO ][o.o.n.Node               ] [wazuh-indexer-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-2778469769811173973, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/wazuh-indexer, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Xms1g, -Xmx1g, -Dlog4j2.formatMsgNoLookups=true, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/usr/share/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2024-10-31T15:43:05,738][INFO ][o.o.s.s.t.SSLConfig      ] [wazuh-indexer-0] SSL dual mode is disabled
[2024-10-31T15:43:05,738][INFO ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] OpenSearch Config path is /usr/share/wazuh-indexer
[2024-10-31T15:43:06,536][INFO ][o.o.s.s.DefaultSecurityKeyStore] [wazuh-indexer-0] JVM supports TLSv1.3
[2024-10-31T15:43:06,619][INFO ][o.o.s.s.DefaultSecurityKeyStore] [wazuh-indexer-0] Config directory is /usr/share/wazuh-indexer/, from there the key- and truststore files are resolved relatively
[2024-10-31T15:43:07,943][INFO ][o.o.s.s.DefaultSecurityKeyStore] [wazuh-indexer-0] TLS Transport Client Provider : JDK
[2024-10-31T15:43:07,944][INFO ][o.o.s.s.DefaultSecurityKeyStore] [wazuh-indexer-0] TLS Transport Server Provider : JDK
[2024-10-31T15:43:07,944][INFO ][o.o.s.s.DefaultSecurityKeyStore] [wazuh-indexer-0] TLS HTTP Provider             : JDK
[2024-10-31T15:43:07,944][INFO ][o.o.s.s.DefaultSecurityKeyStore] [wazuh-indexer-0] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2024-10-31T15:43:07,944][INFO ][o.o.s.s.DefaultSecurityKeyStore] [wazuh-indexer-0] Enabled TLS protocols for HTTP layer      : [TLSv1.2]
[2024-10-31T15:43:08,128][INFO ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] Clustername: wazuh
[2024-10-31T15:43:08,525][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] Directory /usr/share/wazuh-indexer/.cache has insecure file permissions (should be 0700)
[2024-10-31T15:43:08,525][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] Directory /usr/share/wazuh-indexer/.cache/JNA has insecure file permissions (should be 0700)
[2024-10-31T15:43:08,525][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] Directory /usr/share/wazuh-indexer/.cache/JNA/temp has insecure file permissions (should be 0700)
[2024-10-31T15:43:08,526][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] Directory /usr/share/wazuh-indexer/certs has insecure file permissions (should be 0700)
[2024-10-31T15:43:08,526][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/certs/admin.pem has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,526][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/certs/node.pem has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,526][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/certs/admin-key.pem has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,527][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/certs/node-key.pem has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,527][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/certs/root-ca.pem has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,527][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] Directory /usr/share/wazuh-indexer/logs has insecure file permissions (should be 0700)
[2024-10-31T15:43:08,528][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/opensearch.yml has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,528][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/lib/modules has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,528][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/lib/jspawnhelper has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,528][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jarsigner has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,528][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jmod has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,529][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jstatd has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,529][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/java has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,529][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jrunscript has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,529][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jfr has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,530][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jconsole has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,530][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/javadoc has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,530][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/keytool has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,530][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jshell has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,531][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jwebserver has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,531][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/serialver has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,531][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jmap has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,531][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jlink has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,532][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jimage has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,532][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jstack has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,532][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jinfo has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,532][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jar has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,533][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jdeprscan has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,533][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jhsdb has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,533][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jdb has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,533][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/javap has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,533][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jstat has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,534][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jps has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,534][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jpackage has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,534][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jdeps has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,534][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/javac has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,535][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/jcmd has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,535][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/jdk/bin/rmiregistry has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,535][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/performance-analyzer-rca/bin/performance-analyzer-rca has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,535][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/performance-analyzer-rca/bin/performance-analyzer-agent has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,535][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/bin/opensearch-env has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,536][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/bin/indexer-security-init.sh has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,536][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/bin/opensearch-keystore has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,536][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/bin/opensearch-upgrade has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,536][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,619][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/bin/opensearch-performance-analyzer/performance-analyzer-agent has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,619][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/bin/opensearch-cli has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,619][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/bin/systemd-entrypoint has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,619][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/bin/opensearch-shard has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,620][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/bin/opensearch has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,620][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/bin/opensearch-plugin has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,620][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/bin/opensearch-env-from-file has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,620][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/bin/opensearch-node has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,621][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/opensearch-security/internal_users.yml has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,621][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/plugins/opensearch-security/tools/audit_config_migrater.sh has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,621][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/plugins/opensearch-security/tools/securityadmin.sh has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,621][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,621][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-certs-tool.sh has insecure file permissions (should be 0600)
[2024-10-31T15:43:08,622][WARN ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] File /usr/share/wazuh-indexer/plugins/opensearch-security/tools/hash.sh has insecure file permissions (should be 0600)
[2024-10-31T15:43:13,826][INFO ][o.o.p.c.c.PluginSettings ] [wazuh-indexer-0] Trying to create directory /dev/shm/performanceanalyzer/.
[2024-10-31T15:43:13,827][INFO ][o.o.p.c.c.PluginSettings ] [wazuh-indexer-0] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2024-10-31T15:43:15,134][INFO ][o.o.i.r.ReindexPlugin    ] [wazuh-indexer-0] ReindexPlugin reloadSPI called
[2024-10-31T15:43:15,135][INFO ][o.o.i.r.ReindexPlugin    ] [wazuh-indexer-0] Unable to find any implementation for RemoteReindexExtension
[2024-10-31T15:43:15,229][INFO ][o.o.j.JobSchedulerPlugin ] [wazuh-indexer-0] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2024-10-31T15:43:15,318][INFO ][o.o.j.JobSchedulerPlugin ] [wazuh-indexer-0] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2024-10-31T15:43:15,319][INFO ][o.o.j.JobSchedulerPlugin ] [wazuh-indexer-0] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2024-10-31T15:43:15,320][INFO ][o.o.j.JobSchedulerPlugin ] [wazuh-indexer-0] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource
[2024-10-31T15:43:15,321][INFO ][o.o.j.JobSchedulerPlugin ] [wazuh-indexer-0] Loaded scheduler extension: opensearch_sap_job, index: .opensearch-sap--job
[2024-10-31T15:43:15,326][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [aggs-matrix-stats]
[2024-10-31T15:43:15,326][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [analysis-common]
[2024-10-31T15:43:15,326][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [cache-common]
[2024-10-31T15:43:15,326][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [geo]
[2024-10-31T15:43:15,326][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [ingest-common]
[2024-10-31T15:43:15,326][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [ingest-geoip]
[2024-10-31T15:43:15,326][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [ingest-user-agent]
[2024-10-31T15:43:15,327][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [lang-expression]
[2024-10-31T15:43:15,327][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [lang-mustache]
[2024-10-31T15:43:15,327][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [lang-painless]
[2024-10-31T15:43:15,327][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [mapper-extras]
[2024-10-31T15:43:15,327][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [opensearch-dashboards]
[2024-10-31T15:43:15,327][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [parent-join]
[2024-10-31T15:43:15,327][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [percolator]
[2024-10-31T15:43:15,327][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [rank-eval]
[2024-10-31T15:43:15,328][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [reindex]
[2024-10-31T15:43:15,328][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [repository-url]
[2024-10-31T15:43:15,328][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [search-pipeline-common]
[2024-10-31T15:43:15,328][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [systemd]
[2024-10-31T15:43:15,328][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded module [transport-netty4]
[2024-10-31T15:43:15,328][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-alerting]
[2024-10-31T15:43:15,328][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-anomaly-detection]
[2024-10-31T15:43:15,329][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-asynchronous-search]
[2024-10-31T15:43:15,329][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-cross-cluster-replication]
[2024-10-31T15:43:15,329][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-geospatial]
[2024-10-31T15:43:15,329][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-index-management]
[2024-10-31T15:43:15,329][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-job-scheduler]
[2024-10-31T15:43:15,329][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-knn]
[2024-10-31T15:43:15,329][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-ml]
[2024-10-31T15:43:15,329][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-neural-search]
[2024-10-31T15:43:15,330][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-notifications]
[2024-10-31T15:43:15,330][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-notifications-core]
[2024-10-31T15:43:15,330][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-observability]
[2024-10-31T15:43:15,330][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-performance-analyzer]
[2024-10-31T15:43:15,330][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-reports-scheduler]
[2024-10-31T15:43:15,330][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-security]
[2024-10-31T15:43:15,330][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-security-analytics]
[2024-10-31T15:43:15,330][INFO ][o.o.p.PluginsService     ] [wazuh-indexer-0] loaded plugin [opensearch-sql]
[2024-10-31T15:43:15,629][INFO ][o.o.s.OpenSearchSecurityPlugin] [wazuh-indexer-0] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2024-10-31T15:43:15,631][INFO ][o.o.e.ExtensionsManager  ] [wazuh-indexer-0] ExtensionsManager initialized
[2024-10-31T15:43:15,639][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] [wazuh-indexer-0] Using MemorySegmentIndexInput with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
[2024-10-31T15:43:15,724][INFO ][o.o.e.NodeEnvironment    ] [wazuh-indexer-0] using [1] data paths, mounts [[/var/lib/wazuh-indexer (/dev/sdb)]], net usable_space [957.1mb], net total_space [973.4mb], types [ext4]
[2024-10-31T15:43:15,724][INFO ][o.o.e.NodeEnvironment    ] [wazuh-indexer-0] heap size [1gb], compressed ordinary object pointers [true]
[2024-10-31T15:43:15,824][INFO ][o.o.n.Node               ] [wazuh-indexer-0] node name [wazuh-indexer-0], node ID [s2eNY6BwTkyrHQx3lU9PYA], cluster name [wazuh], roles [ingest, remote_cluster_client, data, cluster_manager]
[2024-10-31T15:43:23,635][INFO ][o.o.n.p.NeuralSearch     ] [wazuh-indexer-0] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled]
[2024-10-31T15:43:25,435][WARN ][o.o.s.c.Salt             ] [wazuh-indexer-0] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2024-10-31T15:43:25,537][ERROR][o.o.s.a.s.SinkProvider   ] [wazuh-indexer-0] Default endpoint could not be created, auditlog will not work properly.
[2024-10-31T15:43:25,538][WARN ][o.o.s.a.r.AuditMessageRouter] [wazuh-indexer-0] No default storage available, audit log may not work properly. Please check configuration.
[2024-10-31T15:43:25,538][INFO ][o.o.s.a.i.AuditLogImpl   ] [wazuh-indexer-0] Message routing enabled: false
[2024-10-31T15:43:25,631][INFO ][o.o.s.f.SecurityFilter   ] [wazuh-indexer-0] <NONE> indices are made immutable.
[2024-10-31T15:43:26,536][INFO ][o.o.a.b.ADCircuitBreakerService] [wazuh-indexer-0] Registered memory breaker.
[2024-10-31T15:43:27,722][INFO ][o.o.m.b.MLCircuitBreakerService] [wazuh-indexer-0] Registered ML memory breaker.
[2024-10-31T15:43:27,723][INFO ][o.o.m.b.MLCircuitBreakerService] [wazuh-indexer-0] Registered ML disk breaker.
[2024-10-31T15:43:27,723][INFO ][o.o.m.b.MLCircuitBreakerService] [wazuh-indexer-0] Registered ML native memory breaker.
[2024-10-31T15:43:28,029][INFO ][o.r.Reflections          ] [wazuh-indexer-0] Reflections took 197 ms to scan 1 urls, producing 22 keys and 63 values 
[2024-10-31T15:43:28,322][WARN ][o.o.s.p.SQLPlugin        ] [wazuh-indexer-0] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
karlschriek commented 18 hours ago

Possibly related to this? https://github.com/wazuh/wazuh-kubernetes/pull/839