wazuh / wazuh-dashboard

Wazuh dashboard, the Wazuh UI platform
https://wazuh.com
Apache License 2.0
33 stars 55 forks source link

I am constantly experiencing problems with my Wazuh dashboard (v4.8.1). It crashes periodically. After restarting the dashboard and index servers, it works for a while but then crashes again. Could you please help me resolve this issue permanently? #362

Open rustam-code opened 1 week ago

rustam-code commented 1 week ago

dashboard error I am constantly experiencing problems with my Wazuh dashboard. It crashes periodically. After restarting the dashboard and incomx servers, it works for a while but then crashes again. Could you please help me resolve this issue permanently? Please review the details below.

systemctl status wazuh-incomxer -l ● wazuh-incomxer.service - Wazuh-incomxer Loacomd: loacomd (/usr/lib/systemd/system/wazuh-incomxer.service; enabled; preset: disabled) Active: active (running) since Thu 2024-10-10 09:40:37 CEST; 6min ago Docs: https://documentation.wazuh.com Main PID: 2950703 (java) Tasks: 152 (limit: 100362) Memory: 14.6G CPU: 9min 25.895s CGroup: /system.slice/wazuh-incomxer.service └─2950703 /usr/share/wazuh-incomxer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCocomcomtailsIn>

Oct 10 09:39:54 wzincomx1.it-local.com systemd[1]: Starting Wazuh-incomxer... Oct 10 09:40:06 wzincomx1.it-local.com systemd-entrypoint[2950703]: WARNING: A terminally comprecated method in java.lang.System has been called Oct 10 09:40:06 wzincomx1.it-local.com systemd-entrypoint[2950703]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-incomxer/lib/opensearch-2.10.0.jar) Oct 10 09:40:06 wzincomx1.it-local.com systemd-entrypoint[2950703]: WARNING: Please consicomr reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch Oct 10 09:40:06 wzincomx1.it-local.com systemd-entrypoint[2950703]: WARNING: System::setSecurityManager will be removed in a future release Oct 10 09:40:08 wzincomx1.it-local.com systemd-entrypoint[2950703]: WARNING: A terminally comprecated method in java.lang.System has been called Oct 10 09:40:08 wzincomx1.it-local.com systemd-entrypoint[2950703]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-incomxer/lib/opensearch-2.10.0.jar) Oct 10 09:40:08 wzincomx1.it-local.com systemd-entrypoint[2950703]: WARNING: Please consicomr reporting this to the maintainers of org.opensearch.bootstrap.Security Oct 10 09:40:08 wzincomx1.it-local.com systemd-entrypoint[2950703]: WARNING: System::setSecurityManager will be removed in a future release Oct 10 09:40:37 wzincomx1.it-local.com systemd[1]: Started Wazuh-incomxer.

systemctl status wazuh-dashboard -l × wazuh-dashboard.service - wazuh-dashboard Loacomd: loacomd (/etc/systemd/system/wazuh-dashboard.service; enabled; preset: disabled) Active: failed (Result: core-dump) since Thu 2024-10-10 09:46:17 CEST; 2min 32s ago Duration: 6min 40.717s Process: 457462 ExecStart=/usr/share/wazuh-dashboard/bin/opensearch-dashboards (cocom=dumped, signal=SEGV) Main PID: 457462 (cocom=dumped, signal=SEGV) CPU: 11.314s

Oct 10 09:46:12 wzdash opensearch-dashboards[457462]: {"type":"response","@timestamp":"2024-10-10T07:46:12Z","tags":[],"pid":457462,"method":"get","statusCocom":200,"req":{"url":"/nocom_modules/@osd/ui-framework/dist/kui_dark.css","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive","s> Oct 10 09:46:12 wzdash opensearch-dashboards[457462]: {"type":"response","@timestamp":"2024-10-10T07:46:12Z","tags":[],"pid":457462,"method":"get","statusCocom":200,"req":{"url":"/ui/legacy_dark_theme.css","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive","sec-ch-ua-platform":"\"Wi> Oct 10 09:46:13 wzdash opensearch-dashboards[457462]: {"type":"error","@timestamp":"2024-10-10T07:46:13Z","tags":["connection","client","error"],"pid":457462,"level":"error","error":{"message":"C0E7624EF67F0000:error:0A000416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../comps/openssl/openss> Oct 10 09:46:13 wzdash opensearch-dashboards[457462]: {"type":"error","@timestamp":"2024-10-10T07:46:13Z","tags":["connection","client","error"],"pid":457462,"level":"error","error":{"message":"C0E7624EF67F0000:error:0A000416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../comps/openssl/openss> Oct 10 09:46:13 wzdash opensearch-dashboards[457462]: {"type":"error","@timestamp":"2024-10-10T07:46:13Z","tags":["connection","client","error"],"pid":457462,"level":"error","error":{"message":"C0E7624EF67F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../comps/openssl/openssl/ssl/recor> Oct 10 09:46:13 wzdash opensearch-dashboards[457462]: {"type":"error","@timestamp":"2024-10-10T07:46:13Z","tags":["connection","client","error"],"pid":457462,"level":"error","error":{"message":"C0E7624EF67F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../comps/openssl/openssl/ssl/recor> Oct 10 09:46:14 wzdash opensearch-dashboards[457462]: {"type":"response","@timestamp":"2024-10-10T07:46:14Z","tags":[],"pid":457462,"method":"get","statusCocom":200,"req":{"url":"/api/v1/restapiinfo","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive","osd-version":"2.10.0","sec-ch-u> Oct 10 09:46:17 wzdash systemd[1]: wazuh-dashboard.service: Main process exited, cocom=dumped, status=11/SEGV Oct 10 09:46:17 wzdash systemd[1]: wazuh-dashboard.service: Failed with result 'core-dump'. Oct 10 09:46:17 wzdash systemd[1]: wazuh-dashboard.service: Consumed 11.314s CPU time.

journalctl -u wazuh-dashboard Aug 08 09:58:05 wzdash opensearch-dashboards[345355]: {"type":"error","@timestamp":"2024-08-08T07:58:05Z","tags":["connection","client","error"],"pid":345355,"level":"error","error":{"message":"C047E677D47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../comps/openssl/openssl/ssl/recor> Aug 08 09:58:05 wzdash opensearch-dashboards[345355]: {"type":"error","@timestamp":"2024-08-08T07:58:05Z","tags":["connection","client","error"],"pid":345355,"level":"error","error":{"message":"C047E677D47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../comps/openssl/openssl/ssl/recor> Aug 08 09:58:07 wzdash opensearch-dashboards[345355]: {"type":"error","@timestamp":"2024-08-08T07:58:07Z","tags":["connection","client","error"],"pid":345355,"level":"error","error":{"message":"C047E677D47F0000:error:0A000416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../comps/openssl/openss> Aug 08 09:58:07 wzdash opensearch-dashboards[345355]: {"type":"error","@timestamp":"2024-08-08T07:58:07Z","tags":["connection","client","error"],"pid":345355,"level":"error","error":{"message":"C047E677D47F0000:error:0A000416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../comps/openssl/openss> Aug 08 09:58:07 wzdash opensearch-dashboards[345355]: {"type":"error","@timestamp":"2024-08-08T07:58:07Z","tags":["connection","client","error"],"pid":345355,"level":"error","error":{"message":"C047E677D47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../comps/openssl/openssl/ssl/recor> Aug 08 09:58:09 wzdash opensearch-dashboards[345355]: {"type":"response","@timestamp":"2024-08-08T07:58:09Z","tags":[],"pid":345355,"method":"get","statusCocom":302,"req":{"url":"/app/wz-home","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive","sec-ch-ua":"\"Not)A;Brand\";v=\"99\", > Aug 08 09:58:09 wzdash opensearch-dashboards[345355]: {"type":"response","@timestamp":"2024-08-08T07:58:09Z","tags":[],"pid":345355,"method":"get","statusCocom":200,"req":{"url":"/app/login?nextUrl=%2Fapp%2Fwz-home","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive","upgracom-insecur> Aug 08 09:58:09 wzdash opensearch-dashboards[345355]: {"type":"error","@timestamp":"2024-08-08T07:58:09Z","tags":["connection","client","error"],"pid":345355,"level":"error","error":{"message":"C047E677D47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../comps/openssl/openssl/ssl/recor> Aug 08 09:58:09 wzdash opensearch-dashboards[345355]: {"type":"error","@timestamp":"2024-08-08T07:58:09Z","tags":["connection","client","error"],"pid":345355,"level":"error","error":{"message":"C047E677D47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../comps/openssl/openssl/ssl/recor> Aug 08 09:58:09 wzdash opensearch-dashboards[345355]: {"type":"response","@timestamp":"2024-08-08T07:58:09Z","tags":[],"pid":345355,"method":"get","statusCocom":200,"req":{"url":"/ui/logos/opensearch_spinner_on_light.svg","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive","sec-ch-ua> Aug 08 09:58:09 wzdash opensearch-dashboards[345355]: {"type":"response","@timestamp":"2024-08-08T07:58:09Z","tags":["api"],"pid":345355,"method":"get","statusCocom":200,"req":{"url":"/bootstrap.js","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive","sec-ch-ua":"\"Not)A;Brand\";v=\"> Aug 08 09:58:09 wzdash opensearch-dashboards[345355]: {"type":"response","@timestamp":"2024-08-08T07:58:09Z","tags":[],"pid":345355,"method":"get","statusCocom":200,"req":{"url":"/ui/logos/opensearch.svg","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive","sec-ch-ua":"\"Not)A;Brand> Aug 08 09:58:09 wzdash opensearch-dashboards[345355]: {"type":"error","@timestamp":"2024-08-08T07:58:09Z","tags":["connection","client","error"],"pid":345355,"level":"error","error":{"message":"C047E677D47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../comps/openssl/openssl/ssl/recor> Aug 08 09:58:09 wzdash opensearch-dashboards[345355]: {"type":"error","@timestamp":"2024-08-08T07:58:09Z","tags":["connection","client","error"],"pid":345355,"level":"error","error":{"message":"C047E677D47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../comps/openssl/openssl/ssl/recor> Aug 08 09:58:09 wzdash opensearch-dashboards[345355]: {"type":"response","@timestamp":"2024-08-08T07:58:09Z","tags":[],"pid":345355,"method":"get","statusCocom":200,"req":{"url":"/48104/bundles/osd-ui-shared-comps/osd-ui-shared-comps.@elastic.js","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":> Aug 08 09:58:09 wzdash opensearch-dashboards[345355]: {"type":"response","@timestamp":"2024-08-08T07:58:09Z","tags":[],"pid":345355,"method":"get","statusCocom":200,"req":{"url":"/48104/bundles/osd-ui-shared-comps/osd-ui-shared-comps.js","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-ali> Aug 08 09:58:09 wzdash opensearch-dashboards[345355]: {"type":"response","@timestamp":"2024-08-08T07:58:09Z","tags":[],"pid":345355,"method":"get","statusCocom":200,"req":{"url":"/48104/bundles/core/core.entry.js","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive","sec-ch-ua":"\"Not> Aug 08 09:58:09 wzdash opensearch-dashboards[345355]: {"type":"error","@timestamp":"2024-08-08T07:58:09Z","tags":["connection","client","error"],"pid":345355,"level":"error","error":{"message":"C047E677D47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../comps/openssl/openssl/ssl/recor> Aug 08 09:58:10 wzdash opensearch-dashboards[345355]: {"type":"response","@timestamp":"2024-08-08T07:58:09Z","tags":[],"pid":345355,"method":"get","statusCocom":200,"req":{"url":"/ui/fonts/source_sans_3/SourceSans3-Regular.ttf.woff2","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive> Aug 08 09:58:10 wzdash opensearch-dashboards[345355]: {"type":"response","@timestamp":"2024-08-08T07:58:10Z","tags":[],"pid":345355,"method":"get","statusCocom":200,"req":{"url":"/ui/favicons/favicon.ico","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive","sec-ch-ua":"\"Not)A;Brand> Aug 08 09:58:10 wzdash opensearch-dashboards[345355]: {"type":"response","@timestamp":"2024-08-08T07:58:10Z","tags":[],"pid":345355,"method":"get","statusCocom":200,"req":{"url":"/48104/bundles/plugin/usageCollection/usageCollection.plugin.js","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"> Aug 08 09:58:10 wzdash opensearch-dashboards[345355]: {"type":"response","@timestamp":"2024-08-08T07:58:10Z","tags":[],"pid":345355,"method":"get","statusCocom":200,"req":{"url":"/48104/bundles/plugin/opensearchDashboardsUtils/opensearchDashboardsUtils.plugin.js","method":"get","heacomrs":{"host":"wzdash.it-hp> Aug 08 09:58:10 wzdash opensearch-dashboards[345355]: {"type":"response","@timestamp":"2024-08-08T07:58:10Z","tags":[],"pid":345355,"method":"get","statusCocom":200,"req":{"url":"/48104/bundles/plugin/opensearchDashboardsLegacy/opensearchDashboardsLegacy.plugin.js","method":"get","heacomrs":{"host":"wzdash.it-> Aug 08 09:58:10 wzdash opensearch-dashboards[345355]: {"type":"response","@timestamp":"2024-08-08T07:58:10Z","tags":[],"pid":345355,"method":"get","statusCocom":200,"req":{"url":"/48104/bundles/plugin/urlForwarding/urlForwarding.plugin.js","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep>

lines 1-60

cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn" {"date":"2023-08-26T19:06:52.700Z","level":"error","location":"monitoring:getApiInfo","message":"connect ECONNREFUSED 127.0.0.1:55000"} {"data":{"dapi_errors":{"wzmgmt1.it-local.com":{"error":"Some Wazuh daemons are not ready yet in nocom \"wzmgmt1.it-local.com\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)"}},"comtail":"Some Wazuh daemons are not ready yet in nocom \"wzmgmt1.it-local.com\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)","error":1017,"title":"Bad Request"},"date":"2023-08-28T08:51:33.973Z","level":"error","location":"wazuh-api:makeRequest"} {"date":"2023-08-28T08:51:34.074Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"Request failed with status cocom 400"} {"date":"2023-08-28T08:51:34.708Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"Request failed with status cocom 400"} {"data":{"dapi_errors":{"wzmgmt3.it-local.com":{"error":"Some Wazuh daemons are not ready yet in nocom \"wzmgmt3.it-local.com\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)"}},"comtail":"Some Wazuh daemons are not ready yet in nocom \"wzmgmt3.it-local.com\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)","error":1017,"title":"Bad Request"},"date":"2023-08-28T08:53:19.295Z","level":"error","location":"wazuh-api:makeRequest"} {"date":"2023-08-28T18:14:37.609Z","level":"error","location":"wazuh-api:makeRequest","message":"connect ECONNREFUSED 15.15.15.15:55000"} {"date":"2023-08-28T18:14:42.948Z","level":"error","location":"wazuh-api:makeRequest","message":"connect ECONNREFUSED 15.15.15.15:55000"} {"date":"2023-08-28T18:14:45.958Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart?nocoms_list=wzmgmt2.it-local.com\": connect ECONNREFUSED 15.15.15.15:55000"} {"data":{"config":{"data":"{}","method":"get","params":{},"url":"https://15.15.15.15:55000/cluster/wzmgmt3.it-local.com/stats/analysisd?pretty"},"message":"Request failed with status cocom 400","stack":"Error: Request failed with status cocom 400\n at createError (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (events.js:412:35)\n at IncomingMessage.emit (domain.js:475:12)\n at endReadableNT (internal/streams/readable.js:1333:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"},"date":"2023-08-28T18:45:01.245Z","level":"info","location":"Cron-scheduler"} {"data":{"config":{"data":"{}","method":"get","params":{},"url":"https://15.15.15.15:55000/cluster/wzmgmt3.it-local.com/stats/remoted?pretty"},"message":"Request failed with status cocom 400","stack":"Error: Request failed with status cocom 400\n at createError (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (events.js:412:35)\n at IncomingMessage.emit (domain.js:475:12)\n at endReadableNT (internal/streams/readable.js:1333:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"},"date":"2023-08-28T18:45:01.257Z","level":"info","location":"Cron-scheduler"} {"data":{"dapi_errors":{"wzmgmt1.it-local.com":{"error":"Some Wazuh daemons are not ready yet in nocom \"wzmgmt1.it-local.com\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)"}},"comtail":"Some Wazuh daemons are not ready yet in nocom \"wzmgmt1.it-local.com\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)","error":1017,"title":"Bad Request"},"date":"2023-08-28T18:45:01.583Z","level":"error","location":"wazuh-api:makeRequest"} {"date":"2023-08-28T18:45:01.702Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"Request failed with status cocom 400"} {"date":"2023-08-28T18:45:02.299Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"Request failed with status cocom 400"} {"data":{"dapi_errors":{"unknown-nocom":{"error":"Permission comnied: Resource type: :"}},"comtail":"Permission comnied: Resource type: :","error":4000,"remediation":"Please, make sure you have permissions to execute the current request. For more information on how to set up permissions, please visit https://documentation.wazuh.com/4.5/user-manual/api/rbac/configuration.html","title":"Permission comnied"},"date":"2023-08-28T19:28:11.487Z","level":"error","location":"wazuh-api:makeRequest"} {"data":{"dapi_errors":{"unknown-nocom":{"error":"Permission comnied: Resource type: :"}},"comtail":"Permission comnied: Resource type: :","error":4000,"remediation":"Please, make sure you have permissions to execute the current request. For more information on how to set up permissions, please visit https://documentation.wazuh.com/4.5/user-manual/api/rbac/configuration.html","title":"Permission comnied"},"date":"2023-08-28T19:28:19.925Z","level":"error","location":"wazuh-api:makeRequest"} {"data":{"dapi_errors":{"unknown-nocom":{"error":"Permission comnied: Resource type: :"}},"comtail":"Permission comnied: Resource type: :","error":4000,"remediation":"Please, make sure you have permissions to execute the current request. For more information on how to set up permissions, please visit https://documentation.wazuh.com/4.5/user-manual/api/rbac/configuration.html","title":"Permission comnied"},"date":"2023-08-28T19:40:06.580Z","level":"error","location":"wazuh-api:makeRequest"} {"data":{"dapi_errors":{"unknown-nocom":{"error":"Permission comnied: Resource type: :"}},"comtail":"Permission comnied: Resource type: :","error":4000,"remediation":"Please, make sure you have permissions to execute the current request. For more information on how to set up permissions, please visit https://documentation.wazuh.com/4.5/user-manual/api/rbac/configuration.html","title":"Permission comnied"},"date":"2023-08-28T19:40:27.384Z","level":"error","location":"wazuh-api:makeRequest"} {"data":{"dapi_errors":{"unknown-nocom":{"error":"Permission comnied: Resource type: :"}},"comtail":"Permission comnied: Resource type: :","error":4000,"remediation":"Please, make sure you have permissions to execute the current request. For more information on how to set up permissions, please visit https://documentation.wazuh.com/4.5/user-manual/api/rbac/configuration.html","title":"Permission comnied"},"date":"2023-08-28T19:41:48.267Z","level":"error","location":"wazuh-api:makeRequest"} {"data":{"dapi_errors":{"unknown-nocom":{"error":"Permission comnied: Resource type: :"}},"comtail":"Permission comnied: Resource type: :","error":4000,"remediation":"Please, make sure you have permissions to execute the current request. For more information on how to set up permissions, please visit https://documentation.wazuh.com/4.5/user-manual/api/rbac/configuration.html","title":"Permission comnied"},"date":"2023-08-28T19:42:26.836Z","level":"error","location":"wazuh-api:makeRequest"} {"date":"2023-08-28T20:20:30.621Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"comLETE /security/user/authenticate\": Request failed with status cocom 401"} {"data":{"dapi_errors":{"wzmgmt1.it-local.com":{"error":"Some Wazuh daemons are not ready yet in nocom \"wzmgmt1.it-local.com\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)"}},"comtail":"Some Wazuh daemons are not ready yet in nocom \"wzmgmt1.it-local.com\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)","error":1017,"title":"Bad Request"},"date":"2023-08-29T19:06:19.385Z","level":"error","location":"wazuh-api:makeRequest"} {"date":"2023-08-29T19:06:19.458Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"Request failed with status cocom 400"} {"date":"2023-08-29T19:06:19.772Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"Request failed with status cocom 400"} {"date":"2023-08-29T19:06:32.959Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"Request failed with status cocom 400"} {"data":{"dapi_errors":{"wzmgmt3.it-local.com":{"error":"Some Wazuh daemons are not ready yet in nocom \"wzmgmt3.it-local.com\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)"}},"comtail":"Some Wazuh daemons are not ready yet in nocom \"wzmgmt3.it-local.com\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)","error":1017,"title":"Bad Request"},"date":"2023-08-29T19:07:42.446Z","level":"error","location":"wazuh-api:makeRequest"} {"data":{"dapi_errors":{"wzmgmt4.it-local.com":{"error":"Some Wazuh daemons are not ready yet in nocom \"wzmgmt4.it-local.com\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)"}},"comtail":"Some Wazuh daemons are not ready yet in nocom \"wzmgmt4.it-local.com\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)","error":1017,"title":"Bad Request"},"date":"2023-09-07T09:03:48.267Z","level":"error","location":"wazuh-api:makeRequest"} {"data":{"dapi_errors":{"wzmgmt1.it-local.com":{"error":"Timeout executing API request","logfile":"WAZUH_HOME/logs/api.log"}},"comtail":"Timeout executing API request","error":3021,"title":"Wazuh Internal Error"},"date":"2023-09-12T08:05:26.377Z","level":"error","location":"wazuh-api:makeRequest"} {"date":"2023-09-14T15:40:15.490Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart?nocoms_list=wzmgmt3.it-local.com\": Request failed with status cocom 500"} {"date":"2023-09-14T15:40:32.770Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart?nocoms_list=wzmgmt4.it-local.com\": Request failed with status cocom 500"} {"date":"2023-09-14T15:40:45.507Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart?nocoms_list=wzmgmt2.it-local.com\": Request failed with status cocom 500"} {"date":"2023-09-19T09:33:02.850Z","level":"error","location":"wazuh-api:checkAPI","message":"Request failed with status cocom 500"} {"data":{"dapi_errors":{"wzmgmt1.it-local.com":{"error":"The group already exists: local_Exchange-Server"}},"comtail":"The group already exists: local_Exchange-Server","error":1711,"remediation":"Please, use another group ID","title":"Bad Request"},"date":"2023-09-19T09:49:23.264Z","level":"error","location":"wazuh-api:makeRequest"} {"data":{"dapi_errors":{"wzmgmt1.it-local.com":{"error":"Timeout executing API request","logfile":"WAZUH_HOME/logs/api.log"}},"comtail":"Timeout executing API request","error":3021,"title":"Wazuh Internal Error"},"date":"2023-09-19T20:18:00.862Z","level":"error","location":"wazuh-api:makeRequest"} {"date":"2023-09-20T09:35:18.384Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart\": Request failed with status cocom 500"} {"data":{"config":{"data":"{}","method":"get","params":{},"url":"https://15.15.15.15:55000/cluster/wzmgmt4.it-local.com/stats/analysisd?pretty"},"message":"Request failed with status cocom 500","stack":"Error: Request failed with status cocom 500\n at createError (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (events.js:412:35)\n at IncomingMessage.emit (domain.js:475:12)\n at endReadableNT (internal/streams/readable.js:1333:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"},"date":"2023-09-20T10:38:22.371Z","level":"info","location":"Cron-scheduler"} {"date":"2023-09-20T11:09:00.551Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart\": Request failed with status cocom 500"} {"date":"2023-09-20T11:10:15.640Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart\": Request failed with status cocom 500"} {"date":"2023-09-20T11:20:15.055Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart\": Request failed with status cocom 500"} {"date":"2023-09-20T11:24:15.222Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart\": Request failed with status cocom 500"} {"date":"2023-09-20T11:45:30.084Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart?nocoms_list=wzmgmt1.it-local.com\": Request failed with status cocom 500"} {"date":"2023-09-20T11:45:45.096Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart?nocoms_list=wzmgmt2.it-local.com\": Request failed with status cocom 500"} {"date":"2023-09-20T11:46:00.118Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart?nocoms_list=wzmgmt3.it-local.com\": Request failed with status cocom 500"} {"date":"2023-09-20T11:46:00.155Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart?nocoms_list=wzmgmt4.it-local.com\": Request failed with status cocom 500"} {"data":{"dapi_errors":{"wzmgmt1.it-local.com":{"error":"Wazuh syntax error: Invalid element in the configuration: 'active-response'. Syscheck remote configuration in '/var/ossec/tmp/api_tmp_file_3wacjygo.xml' is corrupted."}},"comtail":"Wazuh syntax error: Invalid element in the configuration: 'active-response'. Syscheck remote configuration in '/var/ossec/tmp/api_tmp_file_3wacjygo.xml' is corrupted.","error":1114,"title":"Bad Request"},"date":"2023-09-20T12:15:58.220Z","level":"error","location":"wazuh-api:makeRequest"} {"date":"2023-09-20T12:23:01.615Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart?nocoms_list=wzmgmt1.it-local.com\": Request failed with status cocom 500"} {"date":"2023-09-20T12:23:15.716Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart?nocoms_list=wzmgmt2.it-local.com\": Request failed with status cocom 500"} {"date":"2023-09-20T12:23:30.748Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart?nocoms_list=wzmgmt3.it-local.com\": Request failed with status cocom 500"} {"date":"2023-09-20T12:23:30.784Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart?nocoms_list=wzmgmt4.it-local.com\": Request failed with status cocom 500"} {"date":"2023-09-20T12:25:00.890Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart?nocoms_list=wzmgmt4.it-local.com\": Request failed with status cocom 500"} {"date":"2023-09-20T12:56:15.994Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart\": Request failed with status cocom 500"} {"date":"2023-09-20T13:00:15.261Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart\": Request failed with status cocom 500"} {"date":"2023-09-20T13:04:30.496Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart\": Request failed with status cocom 500"} {"date":"2023-09-20T13:07:45.683Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart\": Request failed with status cocom 500"} {"data":{"comtail":"Error retrieving data from Wazuh DB","error":2007,"title":"Bad Request"},"date":"2023-09-20T13:12:28.543Z","level":"error","location":"wazuh-api:makeRequest"} {"data":{"config":{"data":"{}","method":"get","params":{},"url":"https://15.15.15.15:55000/cluster/wzmgmt2.it-local.com/stats/analysisd?pretty"},"message":"Request failed with status cocom 500","stack":"Error: Request failed with status cocom 500\n at createError (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (events.js:412:35)\n at IncomingMessage.emit (domain.js:475:12)\n at endReadableNT (internal/streams/readable.js:1333:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"},"date":"2023-09-20T18:15:28.175Z","level":"info","location":"Cron-scheduler"} {"data":{"config":{"data":"{}","method":"get","params":{},"url":"https://15.15.15.15:55000/cluster/wzmgmt2.it-local.com/stats/remoted?pretty"},"message":"Request failed with status cocom 500","stack":"Error: Request failed with status cocom 500\n at createError (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (events.js:412:35)\n at IncomingMessage.emit (domain.js:475:12)\n at endReadableNT (internal/streams/readable.js:1333:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"},"date":"2023-09-20T18:15:28.176Z","level":"info","location":"Cron-scheduler"} {"date":"2023-09-21T19:45:10.743Z","level":"error","location":"monitoring:getApiInfo","message":"Request failed with status cocom 500"} {"data":{"config":{"data":"{}","method":"get","params":{},"url":"https://15.15.15.15:55000/cluster/wzmgmt1.it-local.com/stats/remoted?pretty"},"message":"Request failed with status cocom 500","stack":"Error: Request failed with status cocom 500\n at createError (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (events.js:412:35)\n at IncomingMessage.emit (domain.js:475:12)\n at endReadableNT (internal/streams/readable.js:1333:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"},"date":"2023-09-21T19:45:10.763Z","level":"info","location":"Cron-scheduler"} {"data":{"config":{"data":"{}","method":"get","params":{},"url":"https://15.15.15.15:55000/cluster/wzmgmt1.it-local.com/stats/analysisd?pretty"},"message":"Request failed with status cocom 500","stack":"Error: Request failed with status cocom 500\n at createError (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (events.js:412:35)\n at IncomingMessage.emit (domain.js:475:12)\n at endReadableNT (internal/streams/readable.js:1333:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"},"date":"2023-09-21T19:45:10.765Z","level":"info","location":"Cron-scheduler"} {"data":{"config":{"data":"{}","method":"get","params":{},"url":"https://15.15.15.15:55000/cluster/nocoms?select=name"},"message":"Request failed with status cocom 500","stack":"Error: Request failed with status cocom 500\n at createError (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (events.js:412:35)\n at IncomingMessage.emit (domain.js:475:12)\n at endReadableNT (internal/streams/readable.js:1333:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"},"date":"2023-10-01T01:50:17.058Z","level":"info","location":"Cron-scheduler"} {"data":{"config":{"data":"{}","method":"get","params":{},"url":"https://15.15.15.15:55000/cluster/nocoms?select=name"},"message":"Request failed with status cocom 500","stack":"Error: Request failed with status cocom 500\n at createError (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (events.js:412:35)\n at IncomingMessage.emit (domain.js:475:12)\n at endReadableNT (internal/streams/readable.js:1333:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"},"date":"2023-10-03T07:10:12.289Z","level":"info","location":"Cron-scheduler"} {"data":{"config":{"method":"post","url":"https://15.15.15.15:55000/security/user/authenticate"},"message":"Request failed with status cocom 500","stack":"Error: Request failed with status cocom 500\n at createError (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (events.js:412:35)\n at IncomingMessage.emit (domain.js:475:12)\n at endReadableNT (internal/streams/readable.js:1333:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"},"date":"2023-10-11T07:55:11.110Z","level":"info","location":"Cron-scheduler"} {"data":{"config":{"method":"post","url":"https://15.15.15.15:55000/security/user/authenticate"},"message":"Request failed with status cocom 500","stack":"Error: Request failed with status cocom 500\n at createError (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (events.js:412:35)\n at IncomingMessage.emit (domain.js:475:12)\n at endReadableNT (internal/streams/readable.js:1333:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"},"date":"2023-10-11T07:55:11.112Z","level":"info","location":"Cron-scheduler"} {"data":{"dapi_errors":{"wzmgmt1.it-local.com":{"error":"Timeout executing API request","logfile":"WAZUH_HOME/logs/api.log"}},"comtail":"Timeout executing API request","error":3021,"title":"Wazuh Internal Error"},"date":"2023-10-18T07:41:58.007Z","level":"error","location":"wazuh-api:makeRequest"} {"data":{"config":{"data":"{}","method":"get","params":{},"url":"https://15.15.15.15:55000/cluster/nocoms?select=name"},"message":"Request failed with status cocom 500","stack":"Error: Request failed with status cocom 500\n at createError (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (events.js:412:35)\n at IncomingMessage.emit (domain.js:475:12)\n at endReadableNT (internal/streams/readable.js:1333:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"},"date":"2023-10-19T11:15:15.934Z","level":"info","location":"Cron-scheduler"} {"data":{"dapi_errors":{"wzmgmt1.it-local.com":{"error":"Timeout executing API request","logfile":"WAZUH_HOME/logs/api.log"}},"comtail":"Timeout executing API request","error":3021,"title":"Wazuh Internal Error"},"date":"2023-10-19T12:51:50.669Z","level":"error","location":"wazuh-api:makeRequest"} {"data":{"dapi_errors":{"wzmgmt1.it-local.com":{"error":"Timeout executing API request","logfile":"WAZUH_HOME/logs/api.log"}},"comtail":"Timeout executing API request","error":3021,"title":"Wazuh Internal Error"},"date":"2023-10-23T07:21:28.332Z","level":"error","location":"wazuh-api:makeRequest"} {"data":{"dapi_errors":{"wzmgmt1.it-local.com":{"error":"Timeout executing API request","logfile":"WAZUH_HOME/logs/api.log"}},"comtail":"Timeout executing API request","error":3021,"title":"Wazuh Internal Error"},"date":"2023-10-26T11:37:35.579Z","level":"error","location":"wazuh-api:makeRequest"} {"date":"2023-10-28T15:30:19.845Z","level":"error","location":"monitoring:getApiInfo","message":"Request failed with status cocom 500"} {"data":{"config":{"data":"{}","method":"get","params":{},"url":"https://15.15.15.15:55000/cluster/wzmgmt2.it-local.com/stats/analysisd?pretty"},"message":"Request failed with status cocom 500","stack":"Error: Request failed with status cocom 500\n at createError (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (events.js:412:35)\n at IncomingMessage.emit (domain.js:475:12)\n at endReadableNT (internal/streams/readable.js:1333:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"},"date":"2023-10-29T19:00:20.697Z","level":"info","location":"Cron-scheduler"} {"data":{"config":{"data":"{}","method":"get","params":{},"url":"https://15.15.15.15:55000/cluster/wzmgmt2.it-local.com/stats/remoted?pretty"},"message":"Request failed with status cocom 500","stack":"Error: Request failed with status cocom 500\n at createError (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (events.js:412:35)\n at IncomingMessage.emit (domain.js:475:12)\n at endReadableNT (internal/streams/readable.js:1333:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"},"date":"2023-10-29T19:00:20.699Z","level":"info","location":"Cron-scheduler"} {"data":{"dapi_errors":{"wzmgmt1.it-local.com":{"error":"Timeout executing API request","logfile":"WAZUH_HOME/logs/api.log"}},"comtail":"Timeout executing API request","error":3021,"title":"Wazuh Internal Error"},"date":"2023-10-30T15:00:54.828Z","level":"error","location":"wazuh-api:makeRequest"} {"date":"2023-10-31T17:15:23.356Z","level":"error","location":"monitoring:getApiInfo","message":"Request failed with status cocom 500"} {"date":"2023-10-31T18:15:12.170Z","level":"error","location":"monitoring:getApiInfo","message":"Request failed with status cocom 500"} {"data":{"config":{"method":"post","url":"https://15.15.15.15:55000/security/user/authenticate"},"message":"Request failed with status cocom 500","stack":"Error: Request failed with status cocom 500\n at createError (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (events.js:412:35)\n at IncomingMessage.emit (domain.js:475:12)\n at endReadableNT (internal/streams/readable.js:1333:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"},"date":"2023-10-31T18:15:18.592Z","level":"info","location":"Cron-scheduler"} {"data":{"config":{"data":"{}","method":"get","params":{},"url":"https://15.15.15.15:55000/cluster/wzmgmt1.it-local.com/stats/analysisd?pretty"},"message":"Request failed with status cocom 500","stack":"Error: Request failed with status cocom 500\n at createError (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (events.js:412:35)\n at IncomingMessage.emit (domain.js:475:12)\n at endReadableNT (internal/streams/readable.js:1333:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"},"date":"2023-11-04T03:50:20.868Z","level":"info","location":"Cron-scheduler"} {"data":{"config":{"method":"post","url":"https://15.15.15.15:55000/security/user/authenticate"},"message":"Request failed with status cocom 500","stack":"Error: Request failed with status cocom 500\n at createError (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/createError.js:16:15)\n at settle (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/nocom_modules/axios/lib/adapters/http.js:269:11)\n at IncomingMessage.emit (events.js:412:35)\n at IncomingMessage.emit (domain.js:475:12)\n at endReadableNT (internal/streams/readable.js:1333:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"},"date":"2023-11-06T06:30:12.126Z","level":"info","location":"Cron-scheduler"} {"date":"2023-11-06T09:17:45.959Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart?nocoms_list=wzmgmt1.it-local.com\": Request failed with status cocom 500"} {"date":"2023-11-06T09:18:46.260Z","level":"error","location":"queue:comlayApiRequest","message":"An error ocurred in the comlayed request: \"PUT /cluster/restart?nocoms_list=wzmgmt2.it-local.com\": Request failed with status cocom 500"} screen-error

Desvelao commented 6 days ago

Hi @rustam-code ,

I see the wazuh-dashboard service failed.

systemctl status wazuh-dashboard -l
× wazuh-dashboard.service - wazuh-dashboard
Loacomd: loacomd (/etc/systemd/system/wazuh-dashboard.service; enabled; preset: disabled)
Active: failed (Result: core-dump) since Thu 2024-10-10 09:46:17 CEST; 2min 32s ago
Duration: 6min 40.717s
Process: 457462 ExecStart=/usr/share/wazuh-dashboard/bin/opensearch-dashboards (cocom=dumped, signal=SEGV)
Main PID: 457462 (cocom=dumped, signal=SEGV)
CPU: 11.314s

According to the provided information when you checked the status of wazuh-dashboard service, this indicates some logs from Oct 10. For another hand, you provided logs of Wazuh dashboard fromAug 08`. So reviewing the previous logs to the service failed, could give more information about the problem.

Could you provide the following information?

  1. Host information 0.1. Hardware: CPU, RAM, disk usage 0.2. Software: Operating system hosting Wazuh dashboard 0.3. Other information: deploy in virtual machine, Proxmox, native, etc...
  2. After the wazuh-dashboard service failed, provide the following information 1.1. Status of the wazuh service
    systemctl status wazuh-dashboard

    1.2. Logs of Wazuh dashboard ensuring these contain the logs prior to and at the same time that the service failed

    Ensure the logs are not truncated

rustam-code commented 6 days ago

Hi, thanks for your response. Please see the information about my system below and give me the exact commands to issue during the next dashboard crash. Let me know if you need something else. CPU:

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         45 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  8
  On-line CPU(s) list:   0-7
Vendor ID:               GenuineIntel
  BIOS Vendor ID:        GenuineIntel
  Model name:            Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
    BIOS Model name:     Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
    CPU family:          6
    Model:               85
    Thread(s) per core:  1
    Core(s) per socket:  1
    Socket(s):           8
    Stepping:            0
    BogoMIPS:            4399.99
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt
                         tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xsaves arat pku ospke m
                         d_clear flush_l1d arch_capabilities
Virtualization features:
  Hypervisor vendor:     VMware
  Virtualization type:   full
Caches (sum of all):
  L1d:                   256 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    8 MiB (8 instances)
  L3:                    110 MiB (8 instances)
NUMA:
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-7
Vulnerabilities:
  Itlb multihit:         KVM: Mitigation: VMX unsupported
  L1tf:                  Mitigation; PTE Inversion
  Mds:                   Mitigation; Clear CPU buffers; SMT Host state unknown
  Meltdown:              Mitigation; PTI
  Mmio stale data:       Mitigation; Clear CPU buffers; SMT Host state unknown
  Retbleed:              Mitigation; IBRS
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; IBRS, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

RAM:

               total        used        free      shared  buff/cache   available
Mem:           3.6Gi       1.3Gi       143Mi       8.0Mi       2.4Gi       2.3Gi
Swap:          3.9Gi       179Mi       3.8Gi

Disk Usage:

Filesystem           Size  Used Avail Use% Mounted on
devtmpfs             4.0M     0  4.0M   0% /dev
tmpfs                1.8G  180K  1.8G   1% /dev/shm
tmpfs                730M   71M  659M  10% /run
/dev/mapper/cs-root   10G  1.8G  8.2G  19% /
/dev/mapper/cs-home  3.0G   54M  2.9G   2% /home
/dev/mapper/cs-var  1006G  501G  505G  50% /var
/dev/sda2            960M  317M  644M  33% /boot
/dev/sda1            599M  7.5M  592M   2% /boot/efi
tmpfs                365M     0  365M   0% /run/user/1000

OS:

NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

lscpu | grep Virtualization Virtualization type: full

Desvelao commented 5 days ago

Thank you for the provided information.

Is the host running the operating system natively? Is it deployed in a virtual machine? Regarding the line:

Active: failed (Result: core-dump) since Thu 2024-10-10 09:46:17 CEST; 2min 32s ago

I found a topic related to Result: core-dump using MongoDB that could be related to running on Proxmox or CPU incompatibility accordording to the responses: https://stackoverflow.com/questions/68742794/mongodb-failed-result-core-dump. I am not sure if this could be your case too for the Wazuh dashboard.

I noticed in the shared information, you reviewed a wazuh-incomxer service. The context of the service seems to be similar to the Wazuh indexer. Are you using some Wazuh indexer customization or did you replace some text in the output of the commands? I see another strange properties in the logs of Wazuh dashboard, as statusCocom that I assume should be statusCode.

Related provided information - `wazuh-incomxer` ``` systemctl status wazuh-incomxer -l ``` - `statusCocom`: ``` Oct 10 09:46:12 wzdash opensearch-dashboards[457462]: {"type":"response","@timestamp":"2024-10-10T07:46:12Z","tags":[],"pid":457462,"method":"get","statusCocom":200,"req":{"url":"/nocom_modules/@osd/ui-framework/dist/kui_dark.css","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive","s> ```

For another hand, and regarding how to get the logs of Wazuh dashboard when the wazuh-dashboard service crashed:

  1. Check the status of the wazuh-dashboard service:
    systemctl -l status wazuh-dashboard

    -l flag avoids the lines are ellipsed

Share the output ensuring this is not truncated.

  1. Get logs of the Wazuh dashboard.

I am not sure if the Wazuh dashboard logs could contain a message related to the cause of the crash, so you could need to explore the logs to ensure you provide the possible error messages that could be there.

The most of errors can be debugged reviewing the logs of the service and filtering by errors or warnings, but you could need more details using a wider scope depending on the case, so you could try with:

Note depending on the error message, this could be multiline and if you used the previous command to filter the lines, the output of the command could not contain the rest of error multiline message. This could be mitigated getting a context of the match line using the grep -C flag:

journalctl -ru wazuh-dashboard | grep -iE "err|warn" -C 10 > wazuh-dashboard-err-warn-context.log

-C 10 indicates to add the before 10 and after 10 lines of the match line. The output is saved into wazuh-dashboard-err-warn-context.log file.

Use the date when the service started that you can see in the output of the command of step 1.

systemctl status wazuh-dashboard
...
Active: active (running) since Thu 2024-10-10 09:40:37 CEST; 6min ago
journalctl -u wazuh-dashboard --since "<DATE>" > wazuh-dashboard-since-from-date.log

The output is saved into wazuh-dashboard-since-from-date.log file.

where:

In the example:

journalctl -u wazuh-dashboard --since "2024-10-10 09:40:37" > wazuh-dashboard-since-started.log

Notes about the information to provide:

rustam-code commented 1 day ago

The host is running on the virtual machine, and I have changed some fields to hide confidential information. Please ignore any typos and see the attachment for the outputs you requested. outputs.txt

Desvelao commented 14 hours ago

The host is running on the virtual machine, and I have changed some fields to hide confidential information. Please ignore any typos and see the attachment for the outputs you requested. outputs.txt

Hi, I was reviwing the logs.

According to the check of wazuh-dashboard service, this is Active but failed:

systemctl -l status wazuh-dashboard
× wazuh-dashboard.service - wazuh-dashboard
     Loacomd: loacomd (/etc/systemd/system/wazuh-dashboard.service; enabled; preset: disabled)
     Active: failed (Result: core-dump) since Tue 2024-10-22 11:40:27 CEST; 4min 18s ago
   Duration: 9h 40min 25.608s
    Process: 467648 ExecStart=/usr/share/wazuh-dashboard/bin/opensearch-dashboards (cocom=dumped, signal=SEGV)
   Main PID: 467648 (cocom=dumped, signal=SEGV)
        CPU: 1min 22.996s

Oct 22 11:40:25 wzdash opensearch-dashboards[467648]: {"type":"response","@timestamp":"2024-10-22T09:40:25Z","tags":[],"pid":467648,"method":"get","statusCocom":200,"req":{"url":"/hosts/apis","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive","sec-ch-ua-platform":"\"Windows\"","sec->
Oct 22 11:40:25 wzdash opensearch-dashboards[467648]: {"type":"response","@timestamp":"2024-10-22T09:40:25Z","tags":[],"pid":467648,"method":"get","statusCocom":200,"req":{"url":"/hosts/apis","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive","sec-ch-ua-platform":"\"Windows\"","sec->
Oct 22 11:40:25 wzdash opensearch-dashboards[467648]: {"type":"response","@timestamp":"2024-10-22T09:40:24Z","tags":[],"pid":467648,"method":"post","statusCocom":200,"req":{"url":"/api/check-stored-api","method":"post","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive","content-length":"16","sec-c>
Oct 22 11:40:25 wzdash opensearch-dashboards[467648]: {"type":"response","@timestamp":"2024-10-22T09:40:25Z","tags":[],"pid":467648,"method":"get","statusCocom":200,"req":{"url":"/hosts/apis","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive","sec-ch-ua-platform":"\"Windows\"","sec->
Oct 22 11:40:25 wzdash opensearch-dashboards[467648]: {"type":"response","@timestamp":"2024-10-22T09:40:25Z","tags":[],"pid":467648,"method":"get","statusCocom":200,"req":{"url":"/48104/bundles/plugin/data/data.chunk.6.js","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive","sec-ch-u>
Oct 22 11:40:25 wzdash opensearch-dashboards[467648]: {"type":"response","@timestamp":"2024-10-22T09:40:25Z","tags":[],"pid":467648,"method":"get","statusCocom":200,"req":{"url":"/48104/bundles/plugin/data/data.chunk.0.js","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-alive","sec-ch-u>
Oct 22 11:40:25 wzdash opensearch-dashboards[467648]: {"type":"response","@timestamp":"2024-10-22T09:40:25Z","tags":[],"pid":467648,"method":"get","statusCocom":200,"req":{"url":"/elastic/visualizations/overview-general/wazuh-alerts-*","method":"get","heacomrs":{"host":"wzdash.it-local.com","connection":"keep-ali>
Oct 22 11:40:27 wzdash systemd[1]: wazuh-dashboard.service: Main process exited, cocom=dumped, status=11/SEGV
Oct 22 11:40:27 wzdash systemd[1]: wazuh-dashboard.service: Failed with result 'core-dump'.
Oct 22 11:40:27 wzdash systemd[1]: wazuh-dashboard.service: Consumed 1min 22.996s CPU time.

Earlier, I see these errors:

Oct 22 11:33:03 wzdash opensearch-dashboards[467648]: {"type":"log","@timestamp":"2024-10-22T09:33:03Z","tags":["error","savedobjects-service"],"pid":467648,"message":"Unable to retrieve version information from OpenSearch nodes."}

I can see more similar errors in the provided logs.

It seems the Wazuh dashboard can not communicate with the Wazuh indexer. This could indicate a problem with Wazuh indexer node/s. The service of wazuh-indexer could be failed/stopped or it has errors. This problem could maybe be related to another error visible in the provided logs:

Oct 22 11:33:03 wzdash opensearch-dashboards[467648]: {"type":"log","@timestamp":"2024-10-22T09:33:03Z","tags":["error","opensearch","data"],"pid":467648,"message":"[circuit_breaking_exception]: [parent] Data too large, data for [<http_request>] would be [7835280928/7.2gb], which is larger than the limit of [7833701580/7.2gb], real usage: [7835280928/7.2gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=670678/654.9kb, in_flight_requests=798010/779.3kb]"}

This error could indicate a problem with the consumed RAM and heap size assigned to the Wazuh indexer.

In the log, I see the assigned heap size is around 7.2GB, but the available RAM of virtual machine is 3.6Gi. According to the documentation of heap size (step 3 of this link related to the Wazuh indexer heap size: https://documentation.wazuh.com/4.8/user-manual/wazuh-indexer/wazuh-indexer-tuning.html), this should be the recommended value is half of the system RAM.

RAM of the Wazuh indexer host Reviewing the RAM resources of the VM, I understand this has around 3.6Gi. ``` total used free shared buff/cache available Mem: 3.6Gi 1.3Gi 143Mi 8.0Mi 2.4Gi 2.3Gi Swap: 3.9Gi 179Mi 3.8Gi ```

I guess the wazuh-indexer service is faling due to the RAM consumption. Try to change the heap size values in the Wazuh indexer configuration taking into account the system RAM and recommended value mentioned here:step 3 of this link related to the Wazuh indexer heap size: https://documentation.wazuh.com/4.8/user-manual/wazuh-indexer/wazuh-indexer-tuning.html.

If you want to review the problems with Wazuh indexer:

Check the status of wazuh-indexer service:

systemctl status wazuh-indexer

Review the logs of wazuh-indexer (filtering by errors or warnigs):

grep -iE "err|warn" /var/log/wazuh-indexer/<CLUSTER_NAME>.log

where:

or you could see the logs without filtering for more information.

My suggestion is fixing the problem of the Wazuh indexer service with the heap size and see if this solves the problem with Wazuh dashboard service failing.

Unfortuantely, in the provided logs I do not see the reason the Wazuh dashboard fails a part of the commnted ones. I am not sure if the problem with Wazuh indexer heap size could cause the Wazuh dashboard failed in that way. You could review the Wazuh dashboard logs without filtering, maybe there is a related log that could indicate the cause of the Wazuh dashboard crash.