[BUG] Indexer crashes after a restart command due to plugin load failure

[aghosh0605]

Describe the bug Errors are showing up while loading the plugins after the indexer is restarted

To Reproduce Steps to reproduce the behaviour:

1. systemctl restart wazuh-indexer

Expected behaviour The indexer crashes after using a simple restart command due to a plugin load failure. This issue occurred unexpectedly after a long period without changes or updates, and specific reproducible steps are currently not clear to me.

Plugins

opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-security-analytics
opensearch-sql

Screenshots Logs of the error are attached below.

[2024-10-29T20:25:39,037][INFO ][o.o.p.PluginsService     ] [node-1] loaded plugin [opensearch-reports-scheduler]
[2024-10-29T20:25:39,038][INFO ][o.o.p.PluginsService     ] [node-1] loaded plugin [opensearch-security]
[2024-10-29T20:25:39,039][INFO ][o.o.p.PluginsService     ] [node-1] loaded plugin [opensearch-security-analytics]
[2024-10-29T20:25:39,039][INFO ][o.o.p.PluginsService     ] [node-1] loaded plugin [opensearch-sql]
[2024-10-29T20:25:39,105][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2024-10-29T20:25:39,110][INFO ][o.o.e.ExtensionsManager  ] [node-1] ExtensionsManager initialized
[2024-10-29T20:25:39,134][INFO ][o.o.e.NodeEnvironment    ] [node-1] using [1] data paths, mounts [[/var (/dev/sda7)]], net usable_space [2.5tb], net total_space [2.7tb], types [xfs]
[2024-10-29T20:25:39,135][INFO ][o.o.e.NodeEnvironment    ] [node-1] heap size [20gb], compressed ordinary object pointers [true]
[2024-10-29T20:25:39,940][INFO ][o.o.n.Node               ] [node-1] node name [node-1], node ID [9DD3fUT4S26KIxJSf8wiOQ], cluster name [wazuh-indexer-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
[2024-10-29T20:25:42,096][ERROR][o.o.b.Bootstrap          ] [node-1] Exception
java.lang.IllegalArgumentException: unknown setting [org.opensearch.bootstrap.OpenSearchnode.master] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:608) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:549) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:519) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.common.settings.SettingsModule.<init>(SettingsModule.java:179) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.node.Node.<init>(Node.java:548) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.node.Node.<init>(Node.java:381) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) [opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) [opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.8.0.jar:2.8.0]
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) [opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) [opensearch-2.8.0.jar:2.8.0]
[2024-10-29T20:25:42,098][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [org.opensearch.bootstrap.OpenSearchnode.master] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.8.0.jar:2.8.0]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.8.0.jar:2.8.0]
Caused by: java.lang.IllegalArgumentException: unknown setting [org.opensearch.bootstrap.OpenSearchnode.master] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:608) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:549) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:519) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.common.settings.SettingsModule.<init>(SettingsModule.java:179) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.node.Node.<init>(Node.java:548) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.node.Node.<init>(Node.java:381) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.8.0.jar:2.8.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.8.0.jar:2.8.0]
        ... 6 more

Host/Environment (please complete the following information):

• OS: Ubuntu 20.04.03
• Version 4.7.1

[AlexRuiz7]

Hello @aghosh0605

Looking at the logs, this looks more like a configuration issue. It seems there is a typo in any of the settings preventing the service from starting.

java.lang.IllegalArgumentException: unknown setting [org.opensearch.bootstrap.OpenSearchnode.master] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

Have you performed any configuration changes lately? Have you upgraded wazuh-indexer lately or installed, removed or updated any of the plugins?

Review your opensearch.yml or share with us so we can check it. In that case, remove any sensitive information such as public IPs.

Regards, Alex

[aghosh0605]

Hello @aghosh0605

Looking at the logs, this looks more like a configuration issue. It seems there is a typo in any of the settings preventing the service from starting.

java.lang.IllegalArgumentException: unknown setting [org.opensearch.bootstrap.OpenSearchnode.master] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

Have you performed any configuration changes lately? Have you upgraded wazuh-indexer lately or installed, removed or updated any of the plugins?

Review your opensearch.yml or share with us so we can check it. In that case, remove any sensitive information such as public IPs.

Regards, Alex

Hey, thanks for your support!

Mostly till we remember we hadn't done any changes after setup as it was running fine! I attached the below configuration for opensearch.yml

org.opensearch.bootstrap.OpenSearchnode.master: true
node.data: true
node.ingest: true

cluster.name: wazuh-indexer-cluster
cluster.routing.allocation.disk.threshold_enabled: false

node.max_local_storage_nodes: "3"
path.data: /var/lib/wazuh-indexer
path.logs: /var/log/wazuh-indexer

plugins.security.ssl.http.pemcert_filepath: /etc/wazuh-indexer/certs/node-1.pem
plugins.security.ssl.http.pemkey_filepath: /etc/wazuh-indexer/certs/node-1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/wazuh-indexer/certs/node-1.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/wazuh-indexer/certs/node-1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.ssl.http.enabled_ciphers:
  - "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
  - "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
  - "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
  - "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
plugins.security.ssl.http.enabled_protocols:
  - "TLSv1.2"
plugins.security.authcz.admin_dn:
- "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"

plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

### Option to allow Filebeat-oss 7.10.2 to work ###
compatibility.override_main_response_version: true
node.name: node-1
network.host: 0.0.0.0
cluster.initial_master_nodes: node-1
plugins.security.nodes_dn:
        - CN=node-1,OU=Wazuh,O=Wazuh,L=California,C=US

[AlexRuiz7]

org.opensearch.bootstrap.OpenSearchnode.master: true

That's the unrecognized setting. I don't recognize node.data and node.ingest either.

Wazuh 4.7.1 is based on OpenSearch 2.8.0. I've been reviewing the configuration and some fields are deprecated. These configurations are not included on our builds.

[deprecated] node.master: true
[deprecated] node.data: true
[deprecated] node.ingest: true
[deprecated] node.max_local_storage_nodes: "3" 

Although they are deprecated, they should work. The problem is on the first line, as org.opensearch.bootstrap.OpenSearchnode.master is obviously not a valid setting. It should be node.master: true.

• https://github.com/opensearch-project/OpenSearch/blob/2.8.0/server/src/main/java/org/opensearch/node/Node.java
• https://github.com/opensearch-project/OpenSearch/blob/2.8.0/server/src/main/java/org/opensearch/env/NodeEnvironment.java#L203
• https://github.com/wazuh/wazuh-packages/blob/v4.7.1/stack/indexer/base/files/etc/wazuh-indexer/opensearch.yml

Update: node.max_local_storage_nodesis in fact included in our opensearch.yml. The other 3 settings are not.