Inconsistent results in Vulnerability Detection performance tests

[Rebits]

Description

The Release 4.8.1 - RC 2 - Vulnerability Detection performance test has identified unexpected discrepancies in binary statistics between iterations due to several factors:

• Monitoring Start Times: In v4.8.0, monitoring starts after the vulnerability detection feed process and agent registration have been completed. However, in v4.8.1, this is not the case, leading to variations in the test outcomes.
• Significant Metric Differences: There are substantial differences in monitoring metrics due to minor changes. Specifically, file descriptors and RSS usage have doubled between v4.8.1 and v4.8.0. This discrepancy suggests inconsistent event saturation between tests.

Tasks

• Conduct multiple tests on the same version to determine variance.
• If significant variance is detected between results of the same version, identify the root cause and implement mitigation strategies.
• Investigate the intended start time of the monitoring processes to ensure consistent timing across different iterations.

Validation

• Vulnerability Performance test results are consistent and are a solid method to measure real VD performance among different versions.

[rafabailon]

Pipeline Investigation

Jenkins

The Jenkins Pipeline has several functions for monitoring. They all make calls to Ansible Playbooks.

Pipeline Code: test_cluster.groovy

The monitoring functions are launchManagerMonitor, launchIndexerMonitor and launchDashboardMonitor. The Ansible Playbooks they use are manager_monitor.yaml, indexer_monitor.yaml and dashboard_monitoring.yaml respectively.

Playbooks simply activate the available monitoring tools:

• manager_monitor.yaml
• indexer_monitor.yaml
• dashboard_monitoring.yaml

These are the two Ansible instructions used to activate monitoring:

- name: Start wazuh-metrics monitor
  command: "wazuh-metrics {{ metrics_parameters }}"
  environment:
    PATH: "/usr/local/bin:${PATH}"
  async: "{{ monitor_time }}"
  poll: 0
  when: launch_metrics_monitor

- name: Start wazuh-statistics monitor
  command: "wazuh-statistics {{ stats_parameters }}"
  environment:
    PATH: "/usr/local/bin:${PATH}"
  async: "{{ monitor_time }}"
  poll: 0
  when: launch_stats_monitor

QA

The monitoring that is activated in Ansible Playbooks makes use of wazuh-metrics and wazuh-statistics, which are in the wazuh-qa repository.

• wazuh_statistics.py
• wazuh_metrics.py

For the monitoring itself, there are two Monitor classes depending on what is intended to be monitored.

• binary.py
• statistic.py

There are several options that can be configured. However, these are the only options used in the pipeline.

String metrics_parameters = "-p ${metrics_process_name} -v ${_build_parameters.PKG_VERSION} -s ${monitor_sleep} " +
                            "-u ${data_unit} --store ${_global_parameters['binaries_store_path']}"
String stats_parameters = "-t ${stats_names} -s ${monitor_sleep} --store ${_global_parameters['stats_store_path']}"

This is the meaning of each of the options:

• -p: concatenation of the selected processes in MANAGER_MONITORING_PROCESSES (pipeline parameter)
• -v: PKG_VERSION parameter of the pipeline
• -s: the value of monitor_sleep is always 5
• -u: the value of data_unit is always KB
• --store: the value of 'stats_storepath' is `/home/ec2-user/B${BUILDNUMBER}${JOB_NAME}/stats`
• -t: concatenation of values ​​in MANAGER_MONITORING_STATS (pipeline parameter)

The values ​​that can be modified in the pipeline do not seem to be related to the problem described in the issue. The parameters that could modify the behavior of the monitor have a preset value.

[rafabailon]

On Hold due to Release 4.8.1 - RC 3

[rafabailon]

Pipeline Tests

The next step is to run the pipeline several times and compare the results to see if there are variations.

• First Build (582)

  • Build: CLUSTER-Workload_benchmarks_metrics/582/
  • Artifacts: artifacts.zip

• Second Build (583)

  • Build: CLUSTER-Workload_benchmarks_metrics/583/
  • Artifacts: artifacts.zip

• Third Build (584)

  • Build: CLUSTER-Workload_benchmarks_metrics/584/
  • Artifacts: artifacts.zip

Analysis

• Master

Binary	582	583	584
CPU
RSS

• Worker 1

Binary	582	583	584
CPU
RSS

• Worker 2

Binary	582	583	584
CPU
RSS

Conclusions

I have reviewed the graphs and made some comparative tables to see differences between the three tests carried out. As can be seen in the table, there are no significant differences in monitoring. The graphs have a similar pattern and monitoring begins when indicated by the pipeline.

For example, these are the times in which monitoring begins according to the pipeline:

Monitor	First Build	Second Build	Third Build
Manager	12:11:46 (10:11:46)	13:13:05 (11:13:05)	14:15:08 (12:15:08)
Dashboard	12:11:47 (10:11:47)	13:13:04 (11:13:04)	14:15:07 (12:15:07)
Indexer	12:11:46 (10:11:46)	13:13:04 (11:13:04)	14:15:08 (12:15:08)

The monitor start times match the monitoring start times on the graphs.

[rafabailon]

Example Builds

I have reviewed the example builds in the issue. First I checked the parameters used in the pipeline and they are the same (mostly) that I used in the tests. Due to the tests carried out, I have been able to verify that, with the same parameters, the results achieved are similar.

I have reviewed the monitoring as well. The timestamps that appear in the build match the timestamps on the graphs. The monitoring is activated correctly and begins its activity immediately. In the tests I have carried out the same behavior can be seen.

There does not appear to be a relationship between monitoring and the detected variations.

[MARCOSD4]

LGTM

[juliamagan]

LGTM