search

wazuh / wazuh

Wazuh - The Open Source Security Platform. Unified XDR and SIEM protection for endpoints and cloud workloads.
https://wazuh.com/
Other
11.11k stars 1.68k forks source link

Release 4.10.0 - Beta 1 - Workload benchmarks metrics #26995

Closed javiersanchz closed 2 days ago

javiersanchz commented 4 days ago

The following issue aims to run all workload benchmarks for the current release candidate, report the results, and open new issues for any encountered errors.

Workload benchmarks metrics information
Main release candidate issue https://github.com/wazuh/wazuh/issues/26970
Version 4.10.0
Release candidate # Beta 1
Tag v4.10.0-beta1
Previous Workload benchmarks metrics issue https://github.com/wazuh/wazuh/issues/26811

Test configuration

All tests will be run and workload performance metrics will be obtained for the following clustered environment configurations:
# Agents # Worker nodes
50000 25

Test report procedure

All individual test checks must be marked as:
Pass The test ran successfully.
Xfail The test was expected to fail and it failed. It must be properly justified and reported in an issue.
Skip The test was not run. It must be properly justified and reported in an issue.
Fail The test failed. A new issue must be opened to evaluate and address the problem.
All test results must have one the following statuses:
:green_circle: All checks passed.
:red_circle: There is at least one failed check.
:yellow_circle: There is at least one expected fail or skipped test and no failures.

Any failing test must be properly addressed with a new issue, detailing the error and the possible cause. It must be included in the Fixes section of the current release candidate main issue.

Any expected fail or skipped test must have an issue justifying the reason. All auditors must validate the justification for an expected fail or skipped test.

An extended report of the test results must be attached as a zip or txt. This report can be used by the auditors to dig deeper into any possible failures and details.

Conclusions

All tests have been executed and the results can be found here and here.

All tests have passed and the fails have been reported or justified. I therefore conclude that this issue is finished and OK for this release candidate

Known issues

These issues are known but won't be fixed.

New issues

Auditors validation

The definition of done for this one is the validation of the conclusions and the test results from all auditors.

All checks from below must be accepted in order to close this issue.

javiersanchz commented 4 days ago

API performance tests đŸŸ¡

Artifacts: api_performance_tests.zip

Tests results

Endpoint test name Status Issues Ref.
GET /cluster/local/info :green_circle:
GET /cluster/nodes :green_circle:
GET /cluster/healthcheck :green_circle:
GET /cluster/status :green_circle:
GET /cluster/local/config :green_circle:
GET /cluster/api/config :green_circle:
GET /cluster/configuration/validation :green_circle:
GET /manager/status :green_circle:
GET /manager/info :green_circle:
GET /manager/configuration :green_circle:
GET /manager/logs :green_circle:
GET /manager/api/config :green_circle:
GET /manager/configuration/validation :green_circle:
GET /mitre/groups :green_circle:
GET /mitre/metadata :green_circle:
GET /mitre/mitigations :green_circle:
GET /mitre/references :green_circle:
GET /mitre/software :green_circle:
GET /mitre/tactics :green_circle:
GET /mitre/techniques :green_circle:
GET /overview/agents :green_circle:
GET /tasks/status :green_circle:
PUT /active-response :yellow_circle: https://github.com/wazuh/wazuh-qa/issues/5648 (won't be fixed)
PUT /rootcheck :green_circle:
PUT /syscheck :green_circle:
POST /groups :green_circle:
POST /security/users :green_circle:
POST /security/roles :green_circle:
POST /security/policies :green_circle:
POST /security/rules :green_circle:
PUT /agents/group :yellow_circle: https://github.com/wazuh/wazuh/issues/13872 (won't be fixed)
PUT /agents/group/new_test_group/restart :green_circle:
PUT /agents/restart :green_circle:
POST /agents :green_circle:
POST /agents/insert​ :green_circle:
POST /agents/insert​/quick :green_circle:
GET /agents :green_circle:
GET /agents/no_group :green_circle:
GET /agents/outdated :green_circle:
GET /decoders :green_circle:
GET /groups :green_circle:
GET /lists :green_circle:
GET /rules :green_circle:
GET /security/users :green_circle:
GET /security/roles :green_circle:
GET /security/policies :green_circle:
GET /security/rules :green_circle:
DELETE /groups :green_circle:
DELETE /agents :green_circle:
DELETE /security/users :green_circle:
DELETE /security/roles :green_circle:
DELETE /security/policies :green_circle:
DELETE /security/rules :green_circle:
PUT /manager/restart :green_circle:
PUT /cluster/restart :green_circle:
javiersanchz commented 4 days ago

Cluster

Workload Execution: https://ci.wazuh.info/job/CLUSTER-Workload_benchmarks_metrics/798/artifact/ Artifacts: artifacts.zip

Performance :red_circle:

Manual execution ```console (qa) wazuh@javier:~/Git/wazuh-qa/wazuh-qa/tests/performance/test_cluster$ python3 -m pytest test_cluster_performance --artifacts_path='/home/wazuh/Escritorio/artifacts' --n_workers=25 --n_agents=50000 --html=report.html --self-contained-html =============================================================================================== test session starts =============================================================================================== platform linux -- Python 3.9.16, pytest-7.1.2, pluggy-1.0.0 rootdir: /home/wazuh/Git/wazuh-qa/wazuh-qa/tests, configfile: pytest.ini plugins: metadata-2.0.4, testinfra-5.0.0, html-3.1.1 collected 1 item test_cluster_performance/test_cluster_performance.py F [100%] ==================================================================================================== FAILURES ===================================================================================================== ____________________________________________________________________________________________ test_cluster_performance _____________________________________________________________________________________________ artifacts_path = '/home/wazuh/Escritorio/artifacts', n_workers = 25, n_agents = 50000 def test_cluster_performance(artifacts_path, n_workers, n_agents): """Check that a cluster environment did not exceed certain thresholds. This test obtains various statistics (mean, max, regression coefficient) from CSVs with data generated in a cluster environment (resources used and duration of tasks). These statistics are compared with thresholds established in the data folder. Args: artifacts_path (str): Path where CSVs with cluster information can be found. n_workers (int): Number of workers folders that are expected inside the artifacts path. n_agents (int): Number of agents in the cluster environment. """ if None in (artifacts_path, n_workers, n_agents): pytest.fail("Parameters '--artifacts_path= --n_workers= --n_agents=' are required.") # Check if there are threshold data for the specified number of workers and agents. selected_conf = f"{n_workers}w_{n_agents}a" if selected_conf not in configurations: pytest.fail(f"This is not a supported configuration: {selected_conf}. " f"Supported configurations are: {', '.join(configurations.keys())}.") # Check if path exists and if expected number of workers matches what is found inside artifacts. try: cluster_info = ClusterEnvInfo(artifacts_path).get_all_info() except FileNotFoundError: pytest.fail(f"Path '{artifacts_path}' could not be found or it may not follow the proper structure.") if cluster_info.get('worker_nodes', 0) != int(n_workers): pytest.fail(f"Information of {n_workers} workers was expected inside the artifacts folder, but " f"{cluster_info.get('worker_nodes', 0)} were found.") # Calculate stats from data inside artifacts path. data = {'tasks': ClusterCSVTasksParser(artifacts_path).get_stats(), 'resources': ClusterCSVResourcesParser(artifacts_path).get_stats()} if not data['tasks'] or not data['resources']: pytest.fail(f"Stats could not be retrieved, '{artifacts_path}' path may not exist, it is empty or it may not" f" follow the proper structure.") # Compare each stat with its threshold. for data_name, data_stats in data.items(): for phase, files in data_stats.items(): for file, columns in files.items(): for column, nodes in columns.items(): for node_type, stats in nodes.items(): for stat, value in stats.items(): th_value = configurations[selected_conf][data_name][phase][file][column][node_type][stat] if value[1] >= th_value: exceeded_thresholds.append({'value': value[1], 'threshold': th_value, 'stat': stat, 'column': column, 'node': value[0], 'file': file, 'phase': phase}) try: > assert not exceeded_thresholds, 'Some thresholds were exceeded:\n- ' + '\n- '.join( '{stat} {column} {value} >= {threshold} ({node}, {file}, {phase})'.format(**item) for item in exceeded_thresholds) E AssertionError: Some thresholds were exceeded: E - reg_cof USS(KB) 1288.8672659968279 >= 850 (master, wazuh-clusterd, stable_phase) E - reg_cof FD 0.8511898466419888 >= 0.6 (master, wazuh-clusterd, stable_phase) E assert not [{'column': 'USS(KB)', 'file': 'wazuh-clusterd', 'node': 'master', 'phase': 'stable_phase', ...}, {'column': 'FD', 'file': 'wazuh-clusterd', 'node': 'master', 'phase': 'stable_phase', ...}] test_cluster_performance/test_cluster_performance.py:85: AssertionError ---------------------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------------------- Setup phase took 0:18:22s (2024/11/25 12:41:28 - 2024/11/25 12:59:50). Stable phase took 0:06:33s (2024/11/25 12:59:50 - 2024/11/25 13:06:23). ---------------------------------------------------- generated html file: file:///home/wazuh/Git/wazuh-qa/wazuh-qa/tests/performance/test_cluster/report.html ----------------------------------------------------- ============================================================================================= short test summary info ============================================================================================= FAILED test_cluster_performance/test_cluster_performance.py::test_cluster_performance - AssertionError: Some thresholds were exceeded: ================================================================================================ 1 failed in 0.92s ================================================================================================ ```
Test Status Issues Ref.
test_cluster_performance :red_circle: https://github.com/wazuh/wazuh/issues/27058

Reliability :red_circle:

Manual execution ```console (qa) wazuh@javier:~/Git/wazuh-qa/wazuh-qa/tests/reliability/test_cluster$ python3 -m pytest test_cluster_logs --artifacts_path=/home/wazuh/Escritorio/artifacts =============================================================================================== test session starts =============================================================================================== platform linux -- Python 3.9.16, pytest-7.1.2, pluggy-1.0.0 rootdir: /home/wazuh/Git/wazuh-qa/wazuh-qa/tests, configfile: pytest.ini plugins: metadata-2.0.4, testinfra-5.0.0, html-3.1.1 collected 6 items test_cluster_logs/test_cluster_connection/test_cluster_connection.py F [ 16%] test_cluster_logs/test_cluster_error_logs/test_cluster_error_logs.py F [ 33%] test_cluster_logs/test_cluster_master_logs_order/test_cluster_master_logs_order.py . [ 50%] test_cluster_logs/test_cluster_sync/test_cluster_sync.py . [ 66%] test_cluster_logs/test_cluster_task_order/test_cluster_task_order.py . [ 83%] test_cluster_logs/test_cluster_worker_logs_order/test_cluster_worker_logs_order.py x [100%] ==================================================================================================== FAILURES ===================================================================================================== _____________________________________________________________________________________________ test_cluster_connection _____________________________________________________________________________________________ artifacts_path = '/home/wazuh/Escritorio/artifacts' def test_cluster_connection(artifacts_path): """Verify that no worker disconnects from the master once they are connected. For each worker, this test looks for the first successful connection message in its logs. Then it looks for any failed connection attempts after the successful connection found above. Args: artifacts_path (str): Path where folders with cluster information can be found. """ if not artifacts_path: pytest.fail("Parameter '--artifacts_path=' is required.") cluster_log_files = glob(join(artifacts_path, 'worker_*', 'logs', 'cluster.log')) if len(cluster_log_files) == 0: pytest.fail(f'No files found inside {artifacts_path}.') for log_file in cluster_log_files: with open(log_file) as f: s = mmap(f.fileno(), 0, access=ACCESS_READ) # Search first successful connection message. conn = re.search(rb'^.*Successfully connected to master.*$', s, flags=re.MULTILINE) if not conn: pytest.fail(f'Could not find "Successfully connected to master" message in the ' f'{node_name.search(log_file)[1]}') # Search if there are any connection attempts after the message found above. if re.search(rb'^.*Could not connect to master. Trying.*$|^.*Successfully connected to master.*$', s[conn.end():], flags=re.MULTILINE): disconnected_nodes.append(node_name.search(log_file)[1]) if disconnected_nodes: > pytest.fail(f'The following nodes disconnected from master at any point:\n- ' + '\n- '.join(disconnected_nodes)) E Failed: The following nodes disconnected from master at any point: E - worker_21 E - worker_5 E - worker_1 E - worker_20 E - worker_22 E - worker_12 E - worker_6 E - worker_7 E - worker_14 E - worker_3 E - worker_17 E - worker_8 E - worker_11 E - worker_2 E - worker_4 E - worker_9 E - worker_19 E - worker_16 E - worker_25 E - worker_18 E - worker_15 E - worker_24 E - worker_10 E - worker_23 E - worker_13 test_cluster_logs/test_cluster_connection/test_cluster_connection.py:47: Failed _____________________________________________________________________________________________ test_cluster_error_logs _____________________________________________________________________________________________ artifacts_path = '/home/wazuh/Escritorio/artifacts' def test_cluster_error_logs(artifacts_path): """Look for any error messages in the logs of the cluster nodes. Any error message that is not included in the "white_list" will cause the test to fail. Errors found are attached to an html report if the "--html=report.html" parameter is specified. Args: artifacts_path (str): Path where folders with cluster information can be found. """ if not artifacts_path: pytest.fail('Parameter "--artifacts_path=" is required.') cluster_log_files = glob(join(artifacts_path, '*', 'logs', 'cluster.log')) if len(cluster_log_files) == 0: pytest.fail(f'No files found inside {artifacts_path}.') for log_file in cluster_log_files: with open(log_file) as f: s = mmap(f.fileno(), 0, access=ACCESS_READ) error_lines = re.findall(rb'(^.*?error.*?$)', s, flags=re.MULTILINE | re.IGNORECASE) if error_lines: error_lines = [error for error in error_lines if not error_in_white_list(error)] if error_lines: nodes_with_errors.update({node_name.search(log_file)[1]: error_lines}) > assert not nodes_with_errors, 'Errors were found in the "cluster.log" file of ' \ 'these nodes: \n- ' + '\n- '.join(nodes_with_errors) E AssertionError: Errors were found in the "cluster.log" file of these nodes: E - worker_5 E - worker_1 E - worker_22 E - worker_7 E - master E - worker_10 E assert not {'master': [b"2024/11/25 12:59:47 ERROR: [Master] [D API] Error executing API request locally: 'SecurityError' object ...trics_B798_manager_22] [Main] Connection closed due to an unhandled error: [Errno 104] Connection reset by peer'], ...} test_cluster_logs/test_cluster_error_logs/test_cluster_error_logs.py:57: AssertionError ============================================================================================= short test summary info ============================================================================================= FAILED test_cluster_logs/test_cluster_connection/test_cluster_connection.py::test_cluster_connection - Failed: The following nodes disconnected from master at any point: FAILED test_cluster_logs/test_cluster_error_logs/test_cluster_error_logs.py::test_cluster_error_logs - AssertionError: Errors were found in the "cluster.log" file of these nodes: =============================================================================== 2 failed, 3 passed, 1 xfailed in 186.81s (0:03:06) ================================================================================ ```
Test Status Issues Ref.
test_cluster_connection :green_circle: :
test_cluster_error_logs :red_circle: https://github.com/wazuh/wazuh/issues/27050
test_check_logs_order_workers :green_circle:
test_check_logs_order_master :green_circle:
test_cluster_sync :red_circle: https://github.com/wazuh/wazuh/issues/27052
test_cluster_task_order :green_circle:

Logs :red_circle:

Logs ```console wazuh@javier:~/Descargas$ ./cluster_error_finder.sh Searching for errors in /home/wazuh/Escritorio/artifacts/master/logs/cluster.log: 2024/11/25 12:59:47 ERROR: [Master] [D API] Error executing API request locally: 'SecurityError' object is not subscriptable TypeError: 'SecurityError' object is not subscriptable TypeError: 'SecurityError' object is not subscriptable 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) 2024/11/25 12:59:49 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node "master" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting) ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_10/logs/cluster.log: 2024/11/25 12:59:50 ERROR: [Worker CLUSTER-Workload_benchmarks_metrics_B798_manager_10] [Main] Connection closed due to an unhandled error: [Errno 104] Connection reset by peer 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_11/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_12/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_13/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_14/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_15/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_16/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_17/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_18/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_19/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_1/logs/cluster.log: 2024/11/25 12:59:50 ERROR: [Worker CLUSTER-Workload_benchmarks_metrics_B798_manager_1] [Main] Connection closed due to an unhandled error: [Errno 104] Connection reset by peer 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_20/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_21/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_22/logs/cluster.log: 2024/11/25 12:59:50 ERROR: [Worker CLUSTER-Workload_benchmarks_metrics_B798_manager_22] [Main] Connection closed due to an unhandled error: [Errno 104] Connection reset by peer 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_23/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_24/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_25/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_2/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_3/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_4/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_5/logs/cluster.log: 2024/11/25 12:59:50 ERROR: [Worker CLUSTER-Workload_benchmarks_metrics_B798_manager_5] [Main] Connection closed due to an unhandled error: [Errno 104] Connection reset by peer 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_6/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_7/logs/cluster.log: 2024/11/25 12:59:50 ERROR: [Worker CLUSTER-Workload_benchmarks_metrics_B798_manager_7] [Main] Connection closed due to an unhandled error: [Errno 104] Connection reset by peer 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_8/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- Searching for errors in /home/wazuh/Escritorio/artifacts/worker_9/logs/cluster.log: 2024/11/25 13:00:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:10 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:20 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:30 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:40 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:00:50 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:00 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. 2024/11/25 13:01:43 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds. ------------------------------------------- ```

The errors regarding worker disconnections during the test are expected.

The error regarding "2024/11/25 12:59:48 ERROR: [Master] [D API] Some Wazuh daemons are not ready yet in node 'master' (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)" has already been reported in https://github.com/wazuh/wazuh/issues/27050

fdalmaup commented 3 days ago

Review

Related to the errors found in the execution of the Performance tests, we should check that the same conclusions as in the issue https://github.com/wazuh/wazuh/issues/26840 can be achieved in the current release stage test or if we should open a new issue to carry out the same investigation.

javiersanchz commented 3 days ago

Update

The issue https://github.com/wazuh/wazuh/issues/27058 was created to investigate it.

fdalmaup commented 3 days ago

Review

LGTM