search

wazuh / wazuh-indexer

Wazuh indexer, the Wazuh search engine
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
10 stars 17 forks source link

Warning messages found in the v4.9.0-alpha3 OVA #331

Closed hossam1522 closed 1 month ago

hossam1522 commented 1 month ago

Description

The following log messages have been found in the v4.9.0-alpha3 OVA

/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-24T13:35:18,645][WARN ][o.o.s.c.Salt             ] [node-1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-24T13:35:18,762][WARN ][o.o.s.a.r.AuditMessageRouter] [node-1] No default storage available, audit log may not work properly. Please check configuration.
/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-24T13:35:21,175][WARN ][o.o.s.p.SQLPlugin        ] [node-1] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-24T13:35:23,586][WARN ][o.o.g.DanglingIndicesState] [node-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-24T13:35:25,584][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [node-1] Config override setting update called with empty string. Ignoring.
/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-24T13:35:25,874][WARN ][o.o.o.i.ObservabilityIndex] [node-1] message: index [.opensearch-observability/l-ZIVvMbTXqaJMRzbY8K1w] already exists
/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-24T13:35:25,890][WARN ][o.o.s.SecurityAnalyticsPlugin] [node-1] Failed to initialize LogType config index and builtin log types
/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-24T14:01:07,456][WARN ][o.o.s.c.Salt             ] [node-1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-24T14:01:07,577][WARN ][o.o.s.a.r.AuditMessageRouter] [node-1] No default storage available, audit log may not work properly. Please check configuration.
/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-24T14:01:09,486][WARN ][o.o.s.p.SQLPlugin        ] [node-1] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-24T14:01:11,877][WARN ][o.o.g.DanglingIndicesState] [node-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-24T14:01:14,292][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [node-1] Config override setting update called with empty string. Ignoring.
/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-24T14:01:14,604][WARN ][o.o.o.i.ObservabilityIndex] [node-1] message: index [.opensearch-observability/l-ZIVvMbTXqaJMRzbY8K1w] already exists
/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-24T14:01:14,722][WARN ][o.o.s.SecurityAnalyticsPlugin] [node-1] Failed to initialize LogType config index and builtin log types

They have been found in the previous issues without a final issue reported

We need to validate that these messages are expected or take appropriate action to prevent their occurrence.

Related

f-galland commented 1 month ago

A freshly setup opensearch 2.13 returns the following WARN level logs on /var/log/opensearch/opensearch.log:

[o.o.b.BootstrapChecks    ] [opensearch213] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_cluster_manager_nodes / cluster.initial_master_nodes] must be configured
[o.o.g.DanglingIndicesState] [opensearch213] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [opensearch213] Config override setting update called with empty string. Ignoring.
[o.o.p.c.u.JsonConverter  ] [opensearch213] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[o.o.s.OpenSearchSecurityPlugin] [opensearch213] File /etc/opensearch/securityadmin_demo.sh has insecure file permissions (should be 0600)
[o.o.s.a.r.AuditMessageRouter] [opensearch213] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
[o.o.s.c.Salt             ] [opensearch213] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[o.o.s.p.SQLPlugin        ] [opensearch213] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
f-galland commented 1 month ago

The following WARN level logs are present in the report, but not in the vanilla opensearch logs:

ObservabilityIndex

/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-24T13:35:25,874][WARN ][o.o.o.i.ObservabilityIndex] [node-1] message: index [.opensearch-observability/l-ZIVvMbTXqaJMRzbY8K1w] already exists

This one suggests the index had been created at a prior date. I deployed the Wazuh Indexer to a fresh environment and cannot replicate the error:

root@ova-warnings:~# grep -i observabilityindex /var/log/wazuh-indexer/wazuh-indexer-cluster.log 
[2024-07-26T17:54:55,461][INFO ][o.o.o.i.ObservabilityIndex] [node-1] observability:Index .opensearch-observability creation Acknowledged

SecurityAnalyticsPlugin

/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-24T13:35:25,890][WARN ][o.o.s.SecurityAnalyticsPlugin] [node-1] Failed to initialize LogType config index and builtin log types

It looks like this is a known issue from opensearch:

AuditMessageRouter

/var/log/wazuh-indexer/wazuh-cluster.log:[2024-07-24T14:01:07,577][WARN ][o.o.s.a.r.AuditMessageRouter] [node-1] No default storage available, audit log may not work properly. Please check configuration.

This seems to be a known issue:

f-galland commented 1 month ago

@hossam1522, most of the WARN level logs seem to be either inherited from opensearch or known issues.

The ObservabilityIndex one may be a byproduct of running the installation procedure twice, and I cannot reproduce it in my tests. Can you confirm whether this appears on a fresh install?

Thanks!

hossam1522 commented 1 month ago

Hello @f-galland,

I tried to replicate the ObservabilityIndex WARN level log on a fresh installation but failed. The warning only appears when importing the 4.9.0 OVA into VirtualBox.

gdiazlo commented 1 month ago

Closing as this will be fixed in upstream OpenSearch. The OVA generation process will still produce the warning as the index is probably being created on the OVA creation instead of the start-up.