search

wazuh / wazuh-documentation

Wazuh - Project documentation
https://wazuh.com
191 stars 336 forks source link

Performance Analyzer error after migration #5375

Closed ThepoisonedApple closed 2 years ago

ThepoisonedApple commented 2 years ago

After migration performance analyzer gives error.

[ERROR][o.o.p.r.EventLogFileHandler] [node-2] Error writing entry 'NOT_INITIALIZED'. Cause:
java.nio.file.AccessDeniedException: /dev/shm/performanceanalyzer/1655931595000.tmp

solution:

chown wazuh-indexer:wazuh-indexer /dev/shm/performanceanalyzer/ -R

needs to be added after Migrating to the Wazuh indexer step 8

rauldpm commented 2 years ago

Hello @ThepoisonedApple

Thanks for reporting this, we will investigate it and we will apply a fix accordingly if necessary, could you tell us in which system you are performing the migration?

Regards, Raúl.

ThepoisonedApple commented 2 years ago

system info: OS: Debian19 cluster: all in one Before migration: wazuh-manager 4.2.5-1 opendistroforelasticsearch 1.13.2-1 opendistroforelasticsearch-kibana 1.13.2 filebeat 7.10.2 After Migration: wazuh-manager 4.3.4-1 wazuh-indexer 4.3.4-1 wazuh-dashboard 4.3.4-1 filebeat 7.10.2

rauldpm commented 2 years ago

Update report

By default, the documentation recommends uninstalling the opendistro-perfomance-analyzer plugin and the unattended installation of 4.2 removes the plugin automatically during the install process, it may be the case that the user does a step-by-step and keeps the plugin, so that when starting elasticsearch the following directory is created in /dev/shm/:

drwxr-xr-x. 2 elasticsearch elasticsearch 40 Jun 27 14:24 performanceanalyzer

When performing the migration process, I have reproduced the error in Centos 8 Stream, error in /var/log/wazuh-indexer/wazuh-cluster.log file:

[2022-06-27T14:29:05,022][ERROR][o.o.p.r.EventLogFileHandler] [node-1] Error writing entry 'NOT_INITIALIZED'. Cause:
java.nio.file.AccessDeniedException: /dev/shm/performanceanalyzer/1656340140000.tmp

Conclusion, a note should be added in the migration process to change the owner and group of this directory recursively to wazuh-indexer, as @ThepoisonedApple proposes

More information about the /dev/shm directory: https://datacadamia.com/os/linux/shared_memory

The /dev/shm directory seems to be neccessary to be mounted (tried to start service with shared memory disabled)

Wazuh indexer and Elasticsearch error output Error reported by Wazuh indexer package ``` [2022-06-27T15:12:18,287][INFO ][o.o.p.c.PluginSettings ] [node-1] Trying to create directory /dev/shm/performanceanalyzer/. [2022-06-27T15:12:18,287][ERROR][o.o.p.c.PluginSettings ] [node-1] Invalid metrics location /dev/shm/performanceanalyzer/. Created: false (Expect True), Directory: false (Expect True), CanWrite: false (Expect> [2022-06-27T15:12:18,287][ERROR][o.o.p.c.PluginSettings ] [node-1] Having issue to load all config items. Disabling plugin. org.opensearch.performanceanalyzer.config.ConfigFatalException: Having issue to use path: /dev/shm/performanceanalyzer/ at org.opensearch.performanceanalyzer.config.PluginSettings.validateOrCreateDir(PluginSettings.java:275) ~[performanceanalyzer-rca-1.2.4.0.jar:?] at org.opensearch.performanceanalyzer.config.PluginSettings.loadMetricsLocationFromConfig(PluginSettings.java:253) ~[performanceanalyzer-rca-1.2.4.0.jar:?] at org.opensearch.performanceanalyzer.config.PluginSettings.(PluginSettings.java:196) [performanceanalyzer-rca-1.2.4.0.jar:?] at org.opensearch.performanceanalyzer.config.PluginSettings.createInstance(PluginSettings.java:234) [performanceanalyzer-rca-1.2.4.0.jar:?] at org.opensearch.performanceanalyzer.core.Util.lambda$invokePrivilegedAndLogError$3(Util.java:97) [performanceanalyzer-rca-1.2.4.0.jar:?] at java.security.AccessController.doPrivileged(AccessController.java:312) [?:?] at org.opensearch.performanceanalyzer.core.Util.invokePrivilegedAndLogError(Util.java:93) [performanceanalyzer-rca-1.2.4.0.jar:?] at org.opensearch.performanceanalyzer.config.PluginSettings.(PluginSettings.java:93) [performanceanalyzer-rca-1.2.4.0.jar:?] at org.opensearch.performanceanalyzer.PerformanceAnalyzerPlugin.(PerformanceAnalyzerPlugin.java:183) [opensearch-performance-analyzer-1.2.4.0.jar:1.2.4.0] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) ~[?:?] at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?] at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:726) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:528) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.plugins.PluginsService.(PluginsService.java:194) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.node.Node.(Node.java:396) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.node.Node.(Node.java:319) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-1.2.4.jar:1.2.4] at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-1.2.4.jar:1.2.4] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135) [opensearch-1.2.4.jar:1.2.4] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101) [opensearch-1.2.4.jar:1.2.4] ``` Same error reported by elasticsearch package ``` [2022-06-27T15:18:24,129][INFO ][c.a.o.e.p.c.PluginSettings] [node-1] Trying to create directory /dev/shm/performanceanalyzer/. [2022-06-27T15:18:24,129][ERROR][c.a.o.e.p.c.PluginSettings] [node-1] Invalid metrics location /dev/shm/performanceanalyzer/. Created: false (Expect True), Directory: false (Expect True), CanWrite: false (Expec> [2022-06-27T15:18:24,130][ERROR][c.a.o.e.p.c.PluginSettings] [node-1] Having issue to load all config items. Disabling plugin. com.amazon.opendistro.elasticsearch.performanceanalyzer.config.ConfigFatalException: Having issue to use path: /dev/shm/performanceanalyzer/ at com.amazon.opendistro.elasticsearch.performanceanalyzer.config.PluginSettings.validateOrCreateDir(PluginSettings.java:257) ~[performanceanalyzer-rca-1.13.jar:?] at com.amazon.opendistro.elasticsearch.performanceanalyzer.config.PluginSettings.loadMetricsLocationFromConfig(PluginSettings.java:235) ~[performanceanalyzer-rca-1.13.jar:?] at com.amazon.opendistro.elasticsearch.performanceanalyzer.config.PluginSettings.(PluginSettings.java:180) [performanceanalyzer-rca-1.13.jar:?] at com.amazon.opendistro.elasticsearch.performanceanalyzer.config.PluginSettings.createInstance(PluginSettings.java:218) [performanceanalyzer-rca-1.13.jar:?] at com.amazon.opendistro.elasticsearch.performanceanalyzer.core.Util.lambda$invokePrivilegedAndLogError$3(Util.java:73) [performanceanalyzer-rca-1.13.jar:?] at java.security.AccessController.doPrivileged(AccessController.java:312) [?:?] at com.amazon.opendistro.elasticsearch.performanceanalyzer.core.Util.invokePrivilegedAndLogError(Util.java:69) [performanceanalyzer-rca-1.13.jar:?] at com.amazon.opendistro.elasticsearch.performanceanalyzer.config.PluginSettings.(PluginSettings.java:77) [performanceanalyzer-rca-1.13.jar:?] at com.amazon.opendistro.elasticsearch.performanceanalyzer.metrics.MetricsConfiguration.(MetricsConfiguration.java:42) [performanceanalyzer-rca-1.13.jar:?] at com.amazon.opendistro.elasticsearch.performanceanalyzer.util.Utils.configureMetrics(Utils.java:48) [opendistro-performance-analyzer-1.13.0.0.jar:1.13.0.0] at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.(PerformanceAnalyzerPlugin.java:128) [opendistro-performance-analyzer-1.13.0.0.jar:1.13.0.0] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) ~[?:?] at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?] at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:713) [elasticsearch-7.10.2.jar:7.10.2] at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:658) [elasticsearch-7.10.2.jar:7.10.2] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:479) [elasticsearch-7.10.2.jar:7.10.2] at org.elasticsearch.plugins.PluginsService.(PluginsService.java:168) [elasticsearch-7.10.2.jar:7.10.2] at org.elasticsearch.node.Node.(Node.java:346) [elasticsearch-7.10.2.jar:7.10.2] at org.elasticsearch.node.Node.(Node.java:289) [elasticsearch-7.10.2.jar:7.10.2] at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) [elasticsearch-7.10.2.jar:7.10.2] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) [elasticsearch-7.10.2.jar:7.10.2] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.10.2.jar:7.10.2] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.10.2.jar:7.10.2] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.10.2.jar:7.10.2] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.10.2.jar:7.10.2] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.10.2.jar:7.10.2] at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.10.2.jar:7.10.2] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.10.2.jar:7.10.2] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.10.2.jar:7.10.2] ```

Reported orphan directory if uninstall is performed: https://github.com/wazuh/wazuh-packages/issues/1693

Regards, Raúl.