search

wazuh / wazuh-qa

Wazuh - Quality Assurance
GNU General Public License v2.0
65 stars 32 forks source link

Failed setup_failed_agents test in Vulnerability Detection E2E tests #5665

Closed joaquinsgi closed 2 months ago

joaquinsgi commented 3 months ago
Target version Related issue Related PR/dev branch
4.9.0 RC 1 https://github.com/wazuh/wazuh/issues/25080 4.9.0 beta 1

Description

Analyzing the build: https://ci.wazuh.info/job/Test_e2e_system/342/ of the issue: https://github.com/wazuh/wazuh/issues/25080, an error has been found in test_vulnerability_detector_scans_cases, which appears to be many agents have failed in the setup, in this case agents 1, 4 and 2. Additionally, we can see more issues with missing_vulnerabilities , missing_mitigated_alerts and missing_affected_alerts , which have already been reported:

We can se in the log of the report:

AssertionError: 
E         Test test_vulnerability_detector_scans_cases[upgrade_package_maintain_vulnerability] failed
E         
E         Check no_errors succeeded
E         Check operation_successfull_for_all_agents succeeded
E         Check expected_vulnerabilities_found_in_index failed. Evidences (['missing_vulnerabilities']) can be found in the report.
E         Check no_unexpected_vulnerabilities_found_in_index succeeded
E         Check expected_vulnerability_affected_alert failed. Evidences (['missing_affected_alerts']) can be found in the report.
E         Check expected_vulnerability_mitigated_alert failed. Evidences (['missing_mitigated_alerts']) can be found in the report.
E         Check setup_operation_results failed. Evidences (['setup_failed_agents']) can be found in the report.
E         Check no_duplicated_vulnerabilities succeeded

In setup_failed_agents 

[
    "agent1",
    "agent4",
    "agent2"
]

Proposed checks

Configuration and considerations

Full report: Test_e2e_system_342_test_vulnerability_detector.zip

Expected results

Have no failed agents.

Related

https://github.com/wazuh/wazuh/issues/25080

jnasselle commented 3 months ago

Update

We presume that this issue is related to the following analysis https://github.com/wazuh/wazuh-qa/issues/5655#issuecomment-2292544200

juliamagan commented 3 months ago

After analyzing the results of these two reports:

We have been able to verify that the failures are always the same for the same agents. Both agents are CentOS 7 agents. You can see that the package is installed and detected correctly:

{"timestamp":"2024-08-07T19:21:18.925+0000","rule":{"level":7,"description":"New Yum package installed.","id":"2932","firedtimes":2,"mail":false,"groups":["syslog","yum","config_changed"],"pci_dss":["10.6.1","10.2.7"],"gpg13":["4.10"],"gdpr":["IV_35.7.d"],"hipaa":["164.312.b"],"nist_800_53":["AU.6","AU.14"],"tsc":["CC7.2","CC7.3","CC6.8","CC8.1"]},"agent":{"id":"003","name":"agent1","ip":"172.31.14.234"},"manager":{"name":"ip-172-31-10-80"},"id":"1723058478.5007470","cluster":{"name":"wazuh","node":"master"},"full_log":"Aug 7 19:21:19 ip-172-31-14-234 yum[11752]: Installed: grafana-8.5.5-1.x86_64","predecoder":{"program_name":"yum","timestamp":"Aug 7 19:21:19","hostname":"ip-172-31-14-234"},"decoder":{},"location":"/var/log/messages"}


- Report with timeout fixed:

{"timestamp":"2024-08-15T15:28:11.708+0000","rule":{"level":7,"description":"New Yum package installed.","id":"2932","firedtimes":1,"mail":false,"groups":["syslog","yum","config_changed"],"pci_dss":["10.6.1","10.2.7"],"gpg13":["4.10"],"gdpr":["IV_35.7.d"],"hipaa":["164.312.b"],"nist_800_53":["AU.6","AU.14"],"tsc":["CC7.2","CC7.3","CC6.8","CC8.1"]},"agent":{"id":"001","name":"agent4","ip":"172.31.0.175"},"manager":{"name":"ip-172-31-3-155"},"id":"1723735691.5455360","cluster":{"name":"wazuh","node":"master"},"full_log":"Aug 15 15:28:12 ip-172-31-0-175 yum[17417]: Installed: grafana-8.5.5-1.aarch64","predecoder":{"program_name":"yum","timestamp":"Aug 15 15:28:12","hostname":"ip-172-31-0-175"},"decoder":{},"location":"/var/log/messages"}

{"timestamp":"2024-08-15T15:28:13.020+0000","rule":{"level":7,"description":"New Yum package installed.","id":"2932","firedtimes":2,"mail":false,"groups":["syslog","yum","config_changed"],"pci_dss":["10.6.1","10.2.7"],"gpg13":["4.10"],"gdpr":["IV_35.7.d"],"hipaa":["164.312.b"],"nist_800_53":["AU.6","AU.14"],"tsc":["CC7.2","CC7.3","CC6.8","CC8.1"]},"agent":{"id":"002","name":"agent1","ip":"172.31.10.200"},"manager":{"name":"ip-172-31-3-155"},"id":"1723735693.5457433","cluster":{"name":"wazuh","node":"master"},"full_log":"Aug 15 15:28:14 ip-172-31-10-200 yum[11699]: Installed: grafana-8.5.5-1.x86_64","predecoder":{"program_name":"yum","timestamp":"Aug 15 15:28:14","hostname":"ip-172-31-10-200"},"decoder":{},"location":"/var/log/messages"}


Then the alert of the vulnerability found appears soon after:

- Original report:

{"timestamp":"2024-08-07T19:22:01.024+0000","rule":{"level":10,"description":"CVE-2022-23498 affects grafana","id":"23505","firedtimes":9,"mail":false,"groups":["vulnerability-detector"],"gdpr":["IV_35.7.d"],"pci_dss":["11.2.1","11.2.3"],"tsc":["CC7.1","CC7.2"]},"agent":{"id":"001","name":"agent4","ip":"172.31.11.170"},"manager":{"name":"ip-172-31-10-80"},"id":"1723058521.5039739","cluster":{"name":"wazuh","node":"master"},"decoder":{"name":"json"},"data":{"vulnerability":{"assigner":"GitHub_M","cve":"CVE-2022-23498","cvss":{"cvss3":{"base_score":"7.100000","vector":{"availability":"LOW","confidentiality_impact":"HIGH","integrity_impact":"HIGH","privileges_required":"LOW","scope":"UNCHANGED","user_interaction":"NONE"}}},"cwe_reference":"CWE-200","enumeration":"CVE","package":{"architecture":"aarch64","condition":"Package less than 9.2.10","name":"grafana","source":" ","version":"8.5.5-1"},"published":"2023-02-03T22:15:09Z","rationale":"Grafana is an open-source platform for monitoring and observability. When datasource query caching is enabled, Grafana caches all headers, including grafana_session. As a result, any user that queries a datasource where the caching is enabled can acquire another user’s session. To mitigate the vulnerability you can disable datasource query caching for all datasources. This issue has been patched in versions 9.2.10 and 9.3.4.","reference":"https://github.com/grafana/grafana/security/advisories/GHSA-2j8f-6whh-frc8","severity":"High","status":"Active","title":"CVE-2022-23498 affects grafana","type":"Packages","updated":"2023-11-07T03:44:11Z"}},"location":"vulnerability-detector"}

{"timestamp":"2024-08-07T19:22:01.826+0000","rule":{"level":10,"description":"CVE-2022-23498 affects grafana","id":"23505","firedtimes":19,"mail":false,"groups":["vulnerability-detector"],"gdpr":["IV_35.7.d"],"pci_dss":["11.2.1","11.2.3"],"tsc":["CC7.1","CC7.2"]},"agent":{"id":"003","name":"agent1","ip":"172.31.14.234"},"manager":{"name":"ip-172-31-10-80"},"id":"1723058521.5170577","cluster":{"name":"wazuh","node":"master"},"decoder":{"name":"json"},"data":{"vulnerability":{"assigner":"GitHub_M","cve":"CVE-2022-23498","cvss":{"cvss3":{"base_score":"7.100000","vector":{"availability":"LOW","confidentiality_impact":"HIGH","integrity_impact":"HIGH","privileges_required":"LOW","scope":"UNCHANGED","user_interaction":"NONE"}}},"cwe_reference":"CWE-200","enumeration":"CVE","package":{"architecture":"x86_64","condition":"Package less than 9.2.10","name":"grafana","source":" ","version":"8.5.5-1"},"published":"2023-02-03T22:15:09Z","rationale":"Grafana is an open-source platform for monitoring and observability. When datasource query caching is enabled, Grafana caches all headers, including grafana_session. As a result, any user that queries a datasource where the caching is enabled can acquire another user’s session. To mitigate the vulnerability you can disable datasource query caching for all datasources. This issue has been patched in versions 9.2.10 and 9.3.4.","reference":"https://github.com/grafana/grafana/security/advisories/GHSA-2j8f-6whh-frc8","severity":"High","status":"Active","title":"CVE-2022-23498 affects grafana","type":"Packages","updated":"2023-11-07T03:44:11Z"}},"location":"vulnerability-detector"}


- Report with timeout fixed:

{"timestamp":"2024-08-15T15:28:56.225+0000","rule":{"level":10,"description":"CVE-2022-23498 affects grafana","id":"23505","firedtimes":9,"mail":false,"groups":["vulnerability-detector"],"gdpr":["IV_35.7.d"],"pci_dss":["11.2.1","11.2.3"],"tsc":["CC7.1","CC7.2"]},"agent":{"id":"001","name":"agent4","ip":"172.31.0.175"},"manager":{"name":"ip-172-31-3-155"},"id":"1723735736.5488398","cluster":{"name":"wazuh","node":"master"},"decoder":{"name":"json"},"data":{"vulnerability":{"assigner":"GitHub_M","cve":"CVE-2022-23498","cvss":{"cvss3":{"base_score":"7.100000","vector":{"availability":"LOW","confidentiality_impact":"HIGH","integrity_impact":"HIGH","privileges_required":"LOW","scope":"UNCHANGED","user_interaction":"NONE"}}},"cwe_reference":"CWE-200","enumeration":"CVE","package":{"architecture":"aarch64","condition":"Package less than 9.2.10","name":"grafana","source":" ","version":"8.5.5-1"},"published":"2023-02-03T22:15:09Z","rationale":"Grafana is an open-source platform for monitoring and observability. When datasource query caching is enabled, Grafana caches all headers, including grafana_session. As a result, any user that queries a datasource where the caching is enabled can acquire another user’s session. To mitigate the vulnerability you can disable datasource query caching for all datasources. This issue has been patched in versions 9.2.10 and 9.3.4.","reference":"https://github.com/grafana/grafana/security/advisories/GHSA-2j8f-6whh-frc8","severity":"High","status":"Active","title":"CVE-2022-23498 affects grafana","type":"Packages","updated":"2023-11-07T03:44:11Z"}},"location":"vulnerability-detector"}

{"timestamp":"2024-08-15T15:28:57.737+0000","rule":{"level":10,"description":"CVE-2022-23498 affects grafana","id":"23505","firedtimes":19,"mail":false,"groups":["vulnerability-detector"],"gdpr":["IV_35.7.d"],"pci_dss":["11.2.1","11.2.3"],"tsc":["CC7.1","CC7.2"]},"agent":{"id":"002","name":"agent1","ip":"172.31.10.200"},"manager":{"name":"ip-172-31-3-155"},"id":"1723735737.5619236","cluster":{"name":"wazuh","node":"master"},"decoder":{"name":"json"},"data":{"vulnerability":{"assigner":"GitHub_M","cve":"CVE-2022-23498","cvss":{"cvss3":{"base_score":"7.100000","vector":{"availability":"LOW","confidentiality_impact":"HIGH","integrity_impact":"HIGH","privileges_required":"LOW","scope":"UNCHANGED","user_interaction":"NONE"}}},"cwe_reference":"CWE-200","enumeration":"CVE","package":{"architecture":"x86_64","condition":"Package less than 9.2.10","name":"grafana","source":" ","version":"8.5.5-1"},"published":"2023-02-03T22:15:09Z","rationale":"Grafana is an open-source platform for monitoring and observability. When datasource query caching is enabled, Grafana caches all headers, including grafana_session. As a result, any user that queries a datasource where the caching is enabled can acquire another user’s session. To mitigate the vulnerability you can disable datasource query caching for all datasources. This issue has been patched in versions 9.2.10 and 9.3.4.","reference":"https://github.com/grafana/grafana/security/advisories/GHSA-2j8f-6whh-frc8","severity":"High","status":"Active","title":"CVE-2022-23498 affects grafana","type":"Packages","updated":"2023-11-07T03:44:11Z"}},"location":"vulnerability-detector"}


However, when the test tries to get the indexed vulnerabilities, having spent enough time, it does not find them:

- Original report:

2024-08-07 19:29:39 Vulnerability not found for agent1: Vulnerability(cve='CVE-2022-23498', package_name='grafana', package_version='8.5.5-1', architecture='x86_64') (check_validators.py:30)

2024-08-07 19:29:39 Vulnerability not found for agent4: Vulnerability(cve='CVE-2022-23498', package_name='grafana', package_version='8.5.5-1', architecture='arm64') (check_validators.py:30)


- Report with timeout fixed:

2024-08-15 15:36:39 Vulnerability not found for agent1: Vulnerability(cve='CVE-2022-23498', package_name='grafana', package_version='8.5.5-1', architecture='x86_64') (check_validators.py:30)

2024-08-15 15:36:39 Vulnerability not found for agent4: Vulnerability(cve='CVE-2022-23498', package_name='grafana', package_version='8.5.5-1', architecture='arm64') (check_validators.py:30)

Rebits commented 3 months ago

Firstly, it's important to highlight the failure in the test_consistency_initial_scans case. This indicates that the vulnerability detection process has identified different vulnerabilities for the same agents, which could be due to one of the following reasons, listed from most to least likely:

This failure could be related to other failures of the test, although, in this case the expected alerts were triggered, so this issue seem to be different.

Due to expected alerts appears we can discard a feed download issue, leading into one of the following options:

Currently running the tests manually to determinate the root of this issue.

Rebits commented 3 months ago

I attempted to manually replicate the issue but was unsuccessful. However, I did confirm that Grafana vulnerabilities are appearing in the index, though they seem to lack the architecture details. It's possible that a filter applied to the vulnerabilities index is preventing their detection.

To troubleshoot further, I am currently rerunning the test with debug options enabled.

Rebits commented 3 months ago

The ETA cannot be met because the issue couldn't be replicated, and additional research is necessary. I recommend rescheduling the release for version 4.9.1.

Rebits commented 2 months ago

Vulnerabilities appear correctly in the index, although the test can not collect them. A petition of @davidjiglesias we will close this. For RC1 we will launch these tests with debug options to allow troubleshooting of the issue.