search

wazuh / wazuh-qa

Wazuh - Quality Assurance
GNU General Public License v2.0
61 stars 30 forks source link

Full Yellow Local - Fix Vulnerability Detector tests in 4.2 #1531

Closed damarisg closed 2 years ago

damarisg commented 2 years ago

Description

This issue found while working on Research: Check the integration tests status in 4.2.0-RC8. We detect that Vulnerability Detector Tests aren't working successfully. It's required to investigate the root cause of each Failed. There are a lot of fails, errors, and some warnings.

Test Execution - Results

Manager Local Jenkins Status
Test Execution 1 Results https://devel.ci.wazuh.info/view/Tests/job/Test_integration/882/ 🔴
Test Execution 2 Results https://devel.ci.wazuh.info/view/Tests/job/Test_integration/926/ 🔴
Test Execution 3 Results https://devel.ci.wazuh.info/view/Tests/job/Test_integration/955/ 🔴

Settings

Package Details

Jenkins branch QA branch Version Revision
4.2 1516-4.2.0-full-green 4.2 1.1493
Type Format Architecture Version Revision Tag File name
Server rpm x86_64 4.2.0 40207 v4.2.0-rc8 wazuh-manager-4.2.0-1.1493.x86_64.rpm

Environment

Provider Box OS CPU Memory
Vagrant centos/8 CentOS 8 2 1024

Setup Local_internal_opcion.conf

wazuh_modules.debug=2
monitord.rotate_log=0

Development

In order to finish this issue the following tasks should be fulfilled:

damarisg commented 2 years ago

While the issues were being created, it was possible to see that some methods affect different tests. So when starting to work, you should synchronize with those involved in the different issues that affect the same method.

It could be the case that one applies the fix to an issue and this simultaneously solves the other issues created related to the same method. Or it could also be the case that more research needs to be done on the problem.

I add the list of methods related to some issues, and also a list of issues that need to be investigated for errors.

method Issues Note
make_vuln_callback https://github.com/wazuh/wazuh-qa/issues/1542 - https://github.com/wazuh/wazuh-qa/issues/1565 - https://github.com/wazuh/wazuh-qa/issues/1568
check_feed_imported_successfully https://github.com/wazuh/wazuh-qa/issues/1543 - https://github.com/wazuh/wazuh-qa/issues/1546 - https://github.com/wazuh/wazuh-qa/issues/1553 - https://github.com/wazuh/wazuh-qa/issues/1554
https://github.com/wazuh/wazuh-qa/issues/1551 Case 1: could be an issue related to read_json_file method.
https://github.com/wazuh/wazuh-qa/issues/1552 Case 2: could be an issue related to read_xml_file method.
check_failure_when_importing_feed https://github.com/wazuh/wazuh-qa/issues/1544 - https://github.com/wazuh/wazuh-qa/issues/1547
https://github.com/wazuh/wazuh-qa/issues/1545 Case 1: could be an issue related to read_xml_file method.
Case 2: could be an issue related to check_feed_imported_successfully method.
https://github.com/wazuh/wazuh-qa/issues/1555 Case 2: Issue related to read_xml_file method.
modify_feed con read_xml_file https://github.com/wazuh/wazuh-qa/issues/1548
check_detected_vulnerabilities_number https://github.com/wazuh/wazuh-qa/issues/1566
check_vulnerability_scan_event https://github.com/wazuh/wazuh-qa/issues/1567
Research https://github.com/wazuh/wazuh-qa/issues/1556 - https://github.com/wazuh/wazuh-qa/issues/1557 - https://github.com/wazuh/wazuh-qa/issues/1558 - https://github.com/wazuh/wazuh-qa/issues/1559 - https://github.com/wazuh/wazuh-qa/issues/1560 - https://github.com/wazuh/wazuh-qa/issues/1561 - https://github.com/wazuh/wazuh-qa/issues/1562 - https://github.com/wazuh/wazuh-qa/issues/1563 - https://github.com/wazuh/wazuh-qa/issues/1564
damarisg commented 2 years ago

General Research: Done. It allowed creating each fail contained on this issue.

damarisg commented 2 years ago

Update 19-07-23

New issues were created on this level: Research/Fix:

Issues Created Team
FileMonitor QA
DataBase Bloqued Core
Disable Modules QA

PRs on 1531-full-yellow-vuln-det

PRs PRs
#1604 #1630
#1603 #1617
#1607 #1633
#1631

Tests Affected by Disable Modules

Issue Issue
https://github.com/wazuh/wazuh-qa/issues/1553 https://github.com/wazuh/wazuh-qa/issues/1566
https://github.com/wazuh/wazuh-qa/issues/1554 https://github.com/wazuh/wazuh-qa/issues/1551
https://github.com/wazuh/wazuh-qa/issues/1543 https://github.com/wazuh/wazuh-qa/issues/1552
https://github.com/wazuh/wazuh-qa/issues/1561 https://github.com/wazuh/wazuh-qa/issues/1568
https://github.com/wazuh/wazuh-qa/issues/1569 https://github.com/wazuh/wazuh-qa/issues/1567
https://github.com/wazuh/wazuh-qa/issues/1547 https://github.com/wazuh/wazuh-qa/issues/1548
https://github.com/wazuh/wazuh-qa/issues/1545

Can't to reproduce

The issues that can't reproduce and we estimate that it could fail when all sets of tests are executed.

Issue Issue
https://github.com/wazuh/wazuh-qa/issues/1546 https://github.com/wazuh/wazuh-qa/issues/1564
https://github.com/wazuh/wazuh-qa/issues/1556 https://github.com/wazuh/wazuh-qa/issues/1559
https://github.com/wazuh/wazuh-qa/issues/1557 https://github.com/wazuh/wazuh-qa/issues/1562
https://github.com/wazuh/wazuh-qa/issues/1555

Update 19-07-21

New issues were created on this level: Research/Fix:

Issues Created Team
FileMonitor QA
DataBase Bloqued Core
Disable Modules QA

PRs on 1531-full-yellow-vuln-det

PRs
#1604
#1603
#1607

Tests Affected by Disable Modules

Issue Issue
https://github.com/wazuh/wazuh-qa/issues/1553 https://github.com/wazuh/wazuh-qa/issues/1566
https://github.com/wazuh/wazuh-qa/issues/1554 https://github.com/wazuh/wazuh-qa/issues/1551
https://github.com/wazuh/wazuh-qa/issues/1543 https://github.com/wazuh/wazuh-qa/issues/1552
https://github.com/wazuh/wazuh-qa/issues/1561 https://github.com/wazuh/wazuh-qa/issues/1568
https://github.com/wazuh/wazuh-qa/issues/1569 https://github.com/wazuh/wazuh-qa/issues/1567
https://github.com/wazuh/wazuh-qa/issues/1547 https://github.com/wazuh/wazuh-qa/issues/1548
https://github.com/wazuh/wazuh-qa/issues/1545

Can't to reproduce

The issues that can't reproduce and we estimate that it could fail when all sets of tests are executed.

Issue Issue
https://github.com/wazuh/wazuh-qa/issues/1546 https://github.com/wazuh/wazuh-qa/issues/1564
https://github.com/wazuh/wazuh-qa/issues/1556 https://github.com/wazuh/wazuh-qa/issues/1559
https://github.com/wazuh/wazuh-qa/issues/1557 https://github.com/wazuh/wazuh-qa/issues/1562

Update 16-07-21

New issues were created on this level: Research/Fix:

Issues Created Team
FileMonitor QA
DataBase Bloqued Core
Disable Modules QA

PRs on 1531-full-yellow-vuln-det

PRs
#1604
#1603

Tests Affected by Disable Modules

Issue Issue
https://github.com/wazuh/wazuh-qa/issues/1553 https://github.com/wazuh/wazuh-qa/issues/1566
https://github.com/wazuh/wazuh-qa/issues/1554 https://github.com/wazuh/wazuh-qa/issues/1551
https://github.com/wazuh/wazuh-qa/issues/1543 https://github.com/wazuh/wazuh-qa/issues/1552
https://github.com/wazuh/wazuh-qa/issues/1561 https://github.com/wazuh/wazuh-qa/issues/1568
https://github.com/wazuh/wazuh-qa/issues/1569

Can't to reproduce

The issues that can't reproduce and we estimate that it could fail when all sets of tests are executed.

Issue Issue
https://github.com/wazuh/wazuh-qa/issues/1546 https://github.com/wazuh/wazuh-qa/issues/1564
https://github.com/wazuh/wazuh-qa/issues/1556 https://github.com/wazuh/wazuh-qa/issues/1559
https://github.com/wazuh/wazuh-qa/issues/1557 https://github.com/wazuh/wazuh-qa/issues/1562

Update 15-07-21

A lot of Tests work, when disabling sca, syscollector, and rootcheck. We will add these requirements to execute Vulnerability Detector Tests.

The issues involved that have this approach are: Issue Issue
https://github.com/wazuh/wazuh-qa/issues/1553 https://github.com/wazuh/wazuh-qa/issues/1566
https://github.com/wazuh/wazuh-qa/issues/1554
https://github.com/wazuh/wazuh-qa/issues/1543
https://github.com/wazuh/wazuh-qa/issues/1561

The issues that can't reproduce and we estimate that it could fail when all sets of tests are executed.

Issue Issue
https://github.com/wazuh/wazuh-qa/issues/1546 https://github.com/wazuh/wazuh-qa/issues/1564
https://github.com/wazuh/wazuh-qa/issues/1556 https://github.com/wazuh/wazuh-qa/issues/1559
https://github.com/wazuh/wazuh-qa/issues/1557 https://github.com/wazuh/wazuh-qa/issues/1562

New issues were created on this level: Research/Fix:

Issues Created Team
FileMonitor QA
DataBase Bloqued Core

Requeriments

Update 14-07-21

A lot of Tests work, when disabling sca, syscollector, and rootcheck. We will add these requirements to execute Vulnerability Detector Tests.

The issues involved that have this approach are: Issue
https://github.com/wazuh/wazuh-qa/issues/1553
https://github.com/wazuh/wazuh-qa/issues/1554
https://github.com/wazuh/wazuh-qa/issues/1543

The issues that can't reproduce and we estimate that it could fail when all sets of tests are executed.

Issue Issue
https://github.com/wazuh/wazuh-qa/issues/1546 https://github.com/wazuh/wazuh-qa/issues/1564
https://github.com/wazuh/wazuh-qa/issues/1556 https://github.com/wazuh/wazuh-qa/issues/1559
https://github.com/wazuh/wazuh-qa/issues/1557 https://github.com/wazuh/wazuh-qa/issues/1562
https://github.com/wazuh/wazuh-qa/issues/1555

Requeriments

mdengra commented 2 years ago

2021-07-23

Test group: test_general_settings

Used Wazuh-QA branch: 1531-full-yellow-vuln-det Test results with the default settings in the ossec.conf:

Test Executions Date By Status
test_general_settings_local_r1.log 2021-07-23 Miguel :yellow_circle:
test_general_settings_local_r2.log 2021-07-23 Miguel :red_circle:
test_general_settings_local_r3.log 2021-07-23 Miguel :yellow_circle:

It seems that going from test: test_general_settings_enabled to test: test_general_settings_ignore_time produces the following error:

        # Check if alert does not appear during ignore time
        for _ in range(1, jumps):
            check_time_travel(time_travel=True, interval=timedelta(seconds=seconds_to_travel))
            with pytest.raises(TimeoutError):
                wazuh_log_monitor.start(timeout=get_configuration['metadata']['timeout'],
                                        callback=custom_callback_vulnerability)
>               raise AttributeError('Alert appeared before ignore_time was finished')
E               AttributeError: Alert appeared before ignore_time was finished

test_vulnerability_detector/test_general_settings/test_general_settings_ignore_time.py:88: AttributeError
.
.
.
FAILED test_vulnerability_detector/test_general_settings/test_general_settings_ignore_time.py::test_ignore_time[get_configuration0]

Due to the nature of the error, it may be due to finding messages from the previous test in the current one. I am still investigating.

DProvinciani commented 2 years ago

2021-07-28

Full execution

Used Wazuh-QA branch: 1565-test-cpe-indexing Test results with the default settings in the ossec.conf:

Test Executions Date By Status
test_vulnerability_detector-r1.log 2021-07-28 Diego :red_circle:

Notes: Te execution failed only in the test_scan_results test group. Almost all tests in the group failed except the test_scan_results/test_redhat_duplicate_vulns.py tests.

mdengra commented 2 years ago

2021-07-28

Full execution

Used Wazuh-QA branch: 1565-test-cpe-indexing Test results with the default settings in the ossec.conf:

Test Executions Date By Status
test_vulnerability_detector_local_r1.log 2021-07-27 Miguel :red_circle:
test_vulnerability_detector_local_r2.log 2021-07-27 Miguel :red_circle:
test_vulnerability_detector_local_r3.log 2021-07-27 Miguel :red_circle:

Notes: I confirm what @DProvinciani mentioned, in my testing environments also pass all tests except those in the test_scan_results group.

mdengra commented 2 years ago

2021-07-28

Combinations of tests or groups of tests to locate the failure

Because the first error is logged in the test: test_debian_inventory_debian_feed.py I have made different combinations of tests or groups of tests that are run before the first failure occurs.

Used Wazuh-QA branch: 1565-test-cpe-indexing Test results with the default settings in the ossec.conf:

Test Executions Date By Status
test_general_settings-test_debian_inventory_debian_feed_local_r1.log 2021-07-28 Miguel :yellow_circle:
test_general_settings-test_debian_inventory_debian_feed_local_r2.log 2021-07-28 Miguel :yellow_circle:
test_general_settings-test_debian_inventory_debian_feed_local_r3.log 2021-07-28 Miguel :yellow_circle:
test_providers_update_interval-test_debian_inventory_debian_feed_local_r1.log 2021-07-28 Miguel :yellow_circle:
test_providers_update_interval-test_debian_inventory_debian_feed_local_r2.log 2021-07-28 Miguel :yellow_circle:
test_providers_update_interval-test_debian_inventory_debian_feed_local_r3.log 2021-07-28 Miguel :yellow_circle:
test_providers-test_debian_inventory_debian_feed_local_r1.log 2021-07-28 Miguel :yellow_circle:
test_providers-test_debian_inventory_debian_feed_local_r2.log 2021-07-28 Miguel :yellow_circle:
test_providers-test_debian_inventory_debian_feed_local_r3.log 2021-07-28 Miguel :yellow_circle:
damarisg commented 2 years ago

Update 28-07-21

I divide it in different subgroups to find the errors.

GROUP 1: ( Orden of execution Test Feeds > Tests GeneralSettings > Tests Providers > Tests Scan).

GROUP 2: (Orden of execution Tests GeneralSettings > Tests Providers > Tests Scan).

Test Executions Date By Status
GeneralSetting.log 2021-07-28 Seyla 🟡
Providers.log 2021-07-28 Seyla 🟡 (Fails by a test blocked https://github.com/wazuh/wazuh-qa/issues/1563)
ScanResults.log 2021-07-28 Seyla 🔴 (it allowed reproduce https://github.com/wazuh/wazuh-qa/issues/1568 which closed because it could not reproduce. My theory is that running the test above could break this - this scan run requires investigation)

GROUP 3: ( Orden of execution Tests Providers > Tests Scan).

Test Executions Date By Status
Providers.log 2021-07-28 Seyla 🟡 (Fails by a test blocked https://github.com/wazuh/wazuh-qa/issues/1563)
Scan.log 2021-07-28 Seyla 🔴 (it allowed reproduce https://github.com/wazuh/wazuh-qa/issues/1568 and https://github.com/wazuh/wazuh-qa/issues/1569 which closed because it could not reproduce. - Scan execution requires research)

GROUP 4: ( Vulnerability Detector Complete)

mdengra commented 2 years ago

2021-07-29

Combinations of tests or groups of tests to locate the failure

Because the first error is logged in the test group: test_scan_results I have made different combinations of groups of tests that are run before the first failure occurs.

Test groups: test_general_settings --> test_providers --> test_scan_results Used Wazuh-QA branch: 1531-full-yellow-vuln-det Test results with the default settings in the ossec.conf:

Test Executions Date By Status
test_general_settings-test_providers-test_scan_results_local_r1.log 2021-07-29 Miguel :red_circle:
test_general_settings-test_providers-test_scan_results_local_r2.log 2021-07-29 Miguel :red_circle:
test_general_settings-test_providers-test_scan_results_local_r3.log 2021-07-29 Miguel :red_circle:

The running of this test group fails every time when the first test of the test_scan_results group is tested. That test is test_debian_inventory_debian_feed.py, which is one of the tests listed in the issue: https://github.com/wazuh/wazuh-qa/issues/1635 which is related to issues:

mdengra commented 2 years ago

2021-07-29

As the problem is now delimited, I have re-run the complete tests for the affected groups.

Test groups: test_general_settings --> test_providers --> test_scan_results Used Wazuh-QA branch: 1531-full-yellow-vuln-det Test results with the default settings in the ossec.conf:

Test Executions Date By Status Comment
test_general_settings-test_providers-test_scan_results_local_r1.log 2021-07-29 Miguel :red_circle: 46 of 49 use cases of test_scan_results failed.
test_general_settings-test_providers-test_scan_results_local_r2.log 2021-07-29 Miguel :red_circle: 46 of 49 use cases of test_scan_results failed.
test_general_settings-test_providers-test_scan_results_local_r3.log 2021-07-29 Miguel :red_circle: 46 of 49 use cases of test_scan_results failed.

Checks that have been performed:

damarisg commented 2 years ago

Update 29-07-21

I saw fails on tests Test_Scan_Results for this reason I executed again only this test folder.

Test Executions Date By Status VM
YellowScan.log 2021-07-29 Seyla 🟡 VM-1
YellowScan2.log 2021-07-29 Seyla 🔴 #1569 VM-2
YellowScan3.log 2021-07-29 Seyla 🔴 #1569 VM-3
YellowScan4.log 2021-07-29 Seyla 🟡 VM-4

Approach 1

I execute alone again: #1569 to see if I can reproduce it, but the results are:

Test Executions Date By Status VM
RedhatDuplicateVulns.log 2021-07-29 Seyla 🟡 VM-A
RedhatDuplicateVulns2.log 2021-07-29 Seyla 🟡 VM-B
RedhatDuplicateVulns3.log 2021-07-29 Seyla 🟡 VM-C

I can see that it works successfully, After that I will execute the previus test executed to #1569 Order to launch in the same VM:

  1. test_debian_inventory_debian_feed > 2. test_macos_inventory > 3. test_msu_inventory_msu_feed
  2. test_redhat_duplicate_vulns > 5. test_redhat_inventory_redhat_feed > 6. test_scan_different_cves
  3. test_scan_differents_cves > test_scan_nvd_feed > 8. test_scan_providers_and_nvd > 9. test_ubuntu_inventory_canonical_feed
Test Executions Date By Status
PrimeroDebian.log 2021-07-29 Seyla 🟡
SegundoMacos.log 2021-07-29 Seyla 🟢
TerceroMsuInventory.log 2021-07-29 Seyla 🟢
CuartoRedhatDuplicateVuln.log 2021-07-29 Seyla 🟢
QuintoRedhatInventory.log 2021-07-29 Seyla 🟢
SextoScanDiff.log 2021-07-29 Seyla 🟢
7ScanNVDfeed.log 2021-07-29 Seyla 🟢
8ScanProvNVDfeed.log 2021-07-29 Seyla 🟢
9UbuntuInventory.log 2021-07-29 Seyla 🟢

After to finish with this, execute again tests folder test_scan_results:

Test Executions Date By Status
FinalScan.log 2021-07-29 Seyla 🟢
FinalScan2.log 2021-07-29 Seyla 🟢
FinalScan3.log 2021-07-29 Seyla 🟢

I can't reproduce the error.

Approach 2

Execute again in a new VM to try the reproduce:

After to finish with this, execute again tests folder test_scan_results:

Test Executions Date By Status
DebugScan.log 2021-07-29 Seyla 🔴 https://github.com/wazuh/wazuh/issues/9309
DebugScan2.log 2021-07-29 Seyla 🟢
DebugScan3.log 2021-07-29 Seyla 🟢

I can't reproduce the error.

Approach 3

Execute again tests folder test_scan_results but in differents VMs:

Test Executions Date By Status
ScanVm2.log 2021-07-29 Seyla 🟡
ScanVm3.log 2021-07-29 Seyla 🟡

Now, l execute again full yellow (WIP) to see if I can reproduce it again.

mdengra commented 2 years ago

2021-07-30

I have continued to delimit more the source of the error that blocks the full yellow. I have performed different combinations of the affected tests and groups.

Test groups: test_general_settings --> test_providers --> test_scan_results Used Wazuh-QA branch: 1531-full-yellow-vuln-det Test results with the default settings in the ossec.conf:

Test Executions Date By Status
test_general_settings-even_providers-first_test_scan_results_local_r1.log 2021-07-30 Miguel :yellow_circle:
test_general_settings-odd_providers-first_test_scan_results_local_r1.log 2021-07-30 Miguel :red_circle:
test_general_settings-test_providers_update_from_year-first_test_scan_results.log 2021-07-30 Miguel :yellow_circle:
test_general_settings-test_providers_enabled-first_test_scan_results_local_r1.log 2021-07-30 Miguel :yellow_circle:
test_general_settings-test_providers_no_os-first_test_scan_results_local_r1.log 2021-07-30 Miguel :red_circle:

As we can see, when trying combinations of odd and even tests from the test_providers group, it has been detected that the odd ones cause the problem, so we have been trying each one of them until we found the affected test (test_providers_no_os.py).

To confirm the above, two more tests have been performed:

Test Executions Date By Status
test_general_settings-test_providers_no_os-first_test_scan_results_local_r2.log 2021-07-30 Miguel :red_circle:
test_general_settings-test_providers_no_os-first_test_scan_results_local_r3.log 2021-07-30 Miguel :red_circle:

We also checked if removing the test_general_settings group would cause the failure:

Test Executions Date By Status
test_providers_no_os-first_test_scan_results_local_r1.log 2021-07-30 Miguel :yellow_circle:

As you can see so far, the combination that is causing the error is: test_general_settings/ --> test_providers_no_os.py --> first_test_scan_results/.

Therefore the next thing to do is to check if all the tests in the test_general_settings group or only some of them cause the error:

Test Executions Date By Status
odd_test_general_settings-test_providers_no_os-first_test_scan_results_local_r1.log 2021-07-30 Miguel :yellow_circle:
even_test_general_settings-test_providers_no_os-first_test_scan_results_local_r1.log 2021-07-30 Miguel :red_circle:

In this situation, running the even tests of the test_general_settings group together with test_providers_no_os.py causes problems with the test_scan_results group.

Finally:

Test Executions Date By Status
test_general_settings_run_on_start-test_providers_no_os-first_test_scan_results_local_r1.log 2021-07-30 Miguel :yellow_circle:
test_general_settings_ignore_time-test_providers_no_os-first_test_scan_results_local_r1.log 2021-07-30 Miguel :red_circle:

Therefore, the combination:

test_general_settings_ignore_time.py --> test_providers_no_os.py --> test_scan_results/

It is the one that causes the problem that prevents full_yellow.

damarisg commented 2 years ago

Update 30-07-21

FULL YELLOW - Test Results

Test Executions Date By Status
FullYellowVuln.log 2021-07-30 Seyla 🟡
FullYellowVuln2.log 2021-07-30 Seyla 🟡
FullYellowVuln3.log 2021-07-30 Seyla 🟡

Failes related on :

by:

mdengra commented 2 years ago

2021-07-30

Full execution

Used Wazuh-QA branch: 1531-full-yellow-vuln-det Test results with the default settings in the ossec.conf:

Test Executions Date By Status
test_vulnerability_detector_local_r1.log 2021-07-30 Miguel :yellow_circle:
test_vulnerability_detector_local_r2.log 2021-07-30 Miguel :yellow_circle:
test_vulnerability_detector_local_r3.log 2021-07-30 Miguel :yellow_circle:

Results: Similar to those obtained by @damarisg. Errors in almost all test_scan_results tests (46 of 49 use cases failed).

Failures related on :

by:

DProvinciani commented 2 years ago

2021-07-30

Full execution

Used Wazuh-QA branch: 1531-full-yellow-vuln-det Test results with the default settings in the ossec.conf:

Test Executions Date By Status
test_vulnerability_detector-full_r1.log 2021-07-30 Diego :red_circle:
test_vulnerability_detector-full_r2.log 2021-07-30 Diego :red_circle:
test_vulnerability_detector-full_r3.log 2021-07-30 Diego :red_circle:

Results: The tests show failures in two groups:

We should evaluate whether we consider these rounds as :red_circle: or :yellow_circle: because the failures seem to be expected until the FileMonitor is improved.

damarisg commented 2 years ago

Update 02-08-21

FULL YELLOW - Test Results

Test Executions Date By Status
ResultsVuln.log 02-08-21 Seyla 🔴
ResultsVuln2.log 02-08-21 Seyla 🔴
ResultsVuln3.log 02-08-21 Seyla 🔴

Note: These executions fails for new fails, is required research it.

APPROACH

  1. Reproduce each fails found

damarisg commented 2 years ago

UPDATE 03-08-21

Research - Case 1

After this changes( applied on my branch local), I try to reproduce fail sequence that found Miguel.
test_general_settings/ --> test_providers_no_os.py --> first_test_scan_results/

Test 1 Test 2 Test 3 Date By Status
ResultsCase1Setting.log ResultsCase1ProviderNoOs1.log ResultsCase1Scan.log 03-08-21 Seyla 🟢
ResultsCase1Setting2.log ResultsCase1ProviderNoOs2.log ResultsCase1Scan2.log 03-08-21 Seyla 🟢
ResultsCase1Setting3.log ResultsCase1ProviderNoOs3.log ResultsCase1Scan3.log 03-08-21 Seyla 🟢

test_general_settings_ignore_time.py --> test_providers_no_os.py --> test_scan_results/

Cycle 1

Test 1 Test 2 Date By Status
Cycle2Ignore.log Cycle1ProviderNoOs.log 03-08-21 Seyla 🔴
Cycle1Ignore.log Cycle2ProviderNoOs.log 03-08-21 Seyla 🔴
Cycle3Ignore.log Cycle3ProviderNoOs.log 03-08-21 Seyla 🔴

Research - Case 2

Results Full Vulnerability (with changes of time extended)

Test Executions Date By Status New Fails
FullVulnerability.log 03-08-21 Seyla 🔴 test_vulnerability_detector/test_feeds/test_validate_feed_content.py
FullVulnerability2.log 03-08-21 Seyla 🔴 test_vulnerability_detector/test_feeds/test_validate_feed_content.py
FullVulnerability3.log 03-08-21 Seyla 🔴 test_vulnerability_detector/test_feeds/test_validate_feed_content.py

Research - Case 3

1. Results Full Vulnerability (tier=0)

Test Executions Date By Status
Tier0_FullVulnerability.log 03-08-21 Seyla 🟡
Tier0_FullVulnerability2.log 03-08-21 Seyla 🟡
Tier0_FullVulnerability3.log 03-08-21 Seyla 🟡

2. Results Full Vulnerability (tier=1) - In the same VM that was executed (tier=0)

Test Executions Date By Status
Tier1_FullVulnerability.log 03-08-21 Seyla 🟡
Tier1_FullVulnerability2.log 03-08-21 Seyla 🔴 test_vulnerability_detector/test_windows/test_cpe_indexing.py
Tier1_FullVulnerability3.log 03-08-21 Seyla 🔴 test_vulnerability_detector/test_windows/test_cpe_indexing.py

3. Results Full Vulnerability (tier=1) - In a new VM.

Test Executions Date By Status
Tier1_FullVulnerability_NewVM.log 03-08-21 Seyla 🔴 test_windows/test_cpe_indexing.py test_providers/test_providers_update_interval.py
Tier1_FullVulnerability_NewVM2.log 03-08-21 Seyla 🟡
Tier1_FullVulnerability_NewVM3.log 03-08-21 Seyla 🟡

RESEARCH

  1. test_vulnerability_detector/test_windows/test_cpe_indexing.py
  2. test_providers/test_providers_update_interval.py
  3. test_vulnerability_detector/test_feeds/test_validate_feed_content.py
Status Reference
🔴 new Fails
🟡 Fails expected or pass all with warnings

List Fails Expected affected by https://github.com/wazuh/wazuh/issues/9309 and https://github.com/wazuh/wazuh-qa/issues/1602

pereyra-m commented 2 years ago

04/08/21

All the vulnerability detector tests were run only with Tier 1, every round cleans the environment. These are the results

Test Executions Date By Status
test_vuln_det_tier1_1.log 2021-08-04 Matias :yellow_circle:
test_vuln_det_tier1_2.log 2021-08-04 Matias :yellow_circle:
test_vuln_det_tier1_3.log 2021-08-04 Matias :yellow_circle:
test_vuln_det_tier1_4.log 2021-08-04 Matias :yellow_circle:
test_vuln_det_tier1_5.log 2021-08-04 Matias :yellow_circle:
test_vuln_det_tier1_6.log 2021-08-04 Matias :yellow_circle:
test_vuln_det_tier1_7.log 2021-08-04 Matias :yellow_circle:
test_vuln_det_tier1_8.log 2021-08-04 Matias :yellow_circle:
test_vuln_det_tier1_9.log 2021-08-04 Matias :yellow_circle:
test_vuln_det_tier1_10.log 2021-08-04 Matias :yellow_circle:

All the tests failed, but only the affected ones by the known issues, so the yellow color was used.

Update

All the vulnerability detector tests were run only with Tier 0, every round cleans the environment. These are the results

Test Executions Date By Status
test_vuln_det_tier0_1.log 2021-08-04 Matias :red_circle:
test_vuln_det_tier0_2.log 2021-08-04 Matias :red_circle:
test_vuln_det_tier0_3.log 2021-08-04 Matias :red_circle:
test_vuln_det_tier0_4.log 2021-08-04 Matias :red_circle:
test_vuln_det_tier0_5.log 2021-08-04 Matias :yellow_circle:
test_vuln_det_tier0_6.log 2021-08-04 Matias :yellow_circle:
test_vuln_det_tier0_7.log 2021-08-04 Matias :red_circle:
test_vuln_det_tier0_8.log 2021-08-04 Matias :red_circle:
test_vuln_det_tier0_9.log 2021-08-04 Matias :red_circle:
test_vuln_det_tier0_10.log 2021-08-04 Matias :red_circle:

All the failed tests are due to test_providers/test_providers_multiple_providers.py, only the round 5 and 6 were successful. All these failures are related to connectivity issues, the tests depend on external resources. Here is an example

'Trying to download https://security-tracker.debian.org/tracker/data/json'
DProvinciani commented 2 years ago

04/08/21

All the vulnerability detector tests were run only with Tier 1, every round cleans the environment. These are the results

Test Executions Date By Status
vulnerability_detector_tier1-r1.log 2021-08-04 Diego :yellow_circle:
vulnerability_detector_tier1-r2.log 2021-08-04 Diego :yellow_circle:
vulnerability_detector_tier1-r3.log 2021-08-04 Diego :yellow_circle:

No failures were seen.

damarisg commented 2 years ago

EXECUTION

To conclude, we add a last execution with all details:

Execution by TIERs (VMs clean) - LOCAL

1. Results Test Vulnerability (tier=0)

Test Executions Date By Status
ResultsTier0VulnDet.log 04-08-21 Seyla 🟡
ResultsTier0VulnDet2.log 04-08-21 Seyla 🟡
ResultsTier0_FullVulnDet4.log 04-08-21 Seyla 🟡

2. Results Test Vulnerability (tier=1)

Test Executions Date By Status
Tier1_FullVulnerability.log 04-08-21 Seyla 🟡
Tier1_FullVulnerability2.log 04-08-21 Seyla 🟡
Tier1_FullVulnerability3.log 04-08-21 Seyla 🟡
Tier1_FullVulnerability4.log 04-08-21 Seyla 🟡
Tier1_FullVulnerability5.log 04-08-21 Seyla 🟡

3. Results Test Vulnerability (tier=2)

Test Executions Date By Status
ResultsTier2VulnDet.log 05-08-21 Seyla 🔴
ResultsTier2VulnDet2.log 05-08-21 Seyla 🔴
ResultsTier2VulnDet3.log 05-08-21 Seyla 🔴
ResultsTier2VulnDet4.log 05-08-21 Seyla 🔴

This Fails was checked and we can see that also are affected by wazuh/wazuh#9309 and #1602

4. Results Full Vulnerability

Test Executions Date By Status
ResultsVulnDet.log 05-08-21 Seyla 🔴
ResultsVulnDet2.log 05-08-21 Seyla 🔴
ResultsVulnDet3.log 05-08-21 Seyla 🟡

This Fails was checked and we can see that also are affected by wazuh/wazuh#9309 and #1602

Execution by TIERs - JENKINS

1. Results Full Vulnerability (tier=0)

Test Executions Date By Status
Results 04-08-21 Seyla 🟡
Results1 04-08-21 Seyla 🟡
Results2 04-08-21 Seyla 🟡

2. Results Full Vulnerability (tier=1)

Test Executions Date By Status
Results 05-08-21 Seyla 🟡
Results1 05-08-21 Seyla 🟡
Results2 05-08-21 Seyla 🟡

3. Results Full Vulnerability (tier=2)

Test Executions Date By Status
Results 05-08-21 Seyla 🟡
Results2 05-08-21 Seyla 🟡
Results3 05-08-21 Seyla 🟡

Full Execution - JENKINS

Results Full Vulnerability

Test Executions Date By Status
Results 05-08-21 Seyla 🟡 (It fails but they are due to the issues that have already been reported or blocked, otherwise the rest works. )
Results1 05-08-21 Seyla 🟡(It fails but they are due to the issues that have already been reported or blocked, otherwise the rest works. )
Results2 05-08-21 Seyla 🟡 (It fails but they are due to the issues that have already been reported or blocked, otherwise the rest works. )

List Fails Expected affected by wazuh/wazuh#9309 and #1602

DProvinciani commented 2 years ago

04/08/21

All the vulnerability detector tests now were run only with Tier 0, every round cleans the environment. These are the results

Test Executions Date By Status
vulnerability_detector_tier0-r1.log 2021-08-04 Diego :yellow_circle:
vulnerability_detector_tier0-r2.log 2021-08-04 Diego :yellow_circle:
vulnerability_detector_tier0-r3.log 2021-08-04 Diego :yellow_circle:

No failures were seen.

DProvinciani commented 2 years ago

04/08/21

All the vulnerability detector tests now were run only with Tier 2, every round cleans the environment. These are the results

Test Executions Date By Status
vulnerability_detector_tier2-r1.log 2021-08-04 Diego :yellow_circle:
vulnerability_detector_tier2-r2.log 2021-08-04 Diego :yellow_circle:
vulnerability_detector_tier2-r3.log 2021-08-04 Diego :yellow_circle:

No failures were seen.

DProvinciani commented 2 years ago

05/08/21

All the vulnerability detector tests now were run (full run), every round cleans the environment. These are the results

Test Executions Date By Status
vulnerability_detector_full-r1.log 2021-08-05 Diego :yellow_circle:
vulnerability_detector_full-r2.log 2021-08-05 Diego :yellow_circle:
vulnerability_detector_full-r3.log 2021-08-05 Diego :yellow_circle:

There are failures only in the tests:

These tests are part of the list of tests that are affected by the FileMonitor.

damarisg commented 2 years ago

SUMMARY: STATUS

The tests individually work. When executing the TIER = 2 tests, they generally fail because they don't find the expected logs and give a timeout. Even adding a higher timeout. (It often happens locally).

Now, there is a list that is expected to fail and it is:

Sometimes when the issue occurs: https://github.com/wazuh/wazuh/issues/9309 causes the following tests for around 20min to fail since that is how long it takes to unlock the database.

Each test run individually works. Executions of TIER = 0 and TIER = 1 have a YELLOW as they only have the expected failures affected by the issues mentioned above.

Another thing to note is that Jenkins tests work, except for the ones we expect to fail.

FileMonitor issue causes there to be inconsistencies, we know that there are some constants but new ones may arise as the last executions of TIER 2.

That depends on a rework of the tests, FileMonitor (Queue) and the improvements of the Framework.

pereyra-m commented 2 years ago

05/08/21

All the vulnerability detector tests were run (no tier filter). These are the results

Test Executions Date By Status
test_vuln_det_all_tier_1.log 2021-08-05 Matias :red_circle: test_providers_multiple_providers
test_vuln_det_all_tier_2.log 2021-08-05 Matias :yellow_circle:
test_vuln_det_all_tier_3.log 2021-08-05 Matias :yellow_circle:

All the tests failed, but the red circle is for the failed tests that are not present in the list. These are related to connectivity issues, like the following

TimeoutError: Event 'Trying to download https://security-tracker.debian.org/tracker/data/json' was not received
...
TimeoutError: Event 'Trying to download file:///home/vagrant/wazuh-qa/tests/integration/test_vulnerability_detector/test_providers/../data/feeds/custom_debian_json_feed.json' was not received
DProvinciani commented 2 years ago

08/09/21

All the Vulnerability Detector tests were run after merging the fix in #1687. All the failures were related to the FileMonitor in both test_providers/test_providers_no_os and test_scan_results tests.

Test Executions Date By Status
test_vulnerability_detector-full-r1.log 2021-08-06 Diego :yellow_circle:
test_vulnerability_detector-full-r2.log 2021-08-06 Diego :yellow_circle:
test_vulnerability_detector-full-r3.log 2021-08-06 Diego :yellow_circle:
pereyra-m commented 2 years ago

08/09/21

All the Vulnerability Detector tests were run again, after merging the fix in #1687. All the rounds failed, and only the number 2 is yellow because all the failed tests are in the known list.

Test Executions Date By Status
test_vuln_det_all_tier_1.log 2021-08-09 Matias :red_circle: test_general_settings_ignore_time and test_providers_multiple_providers
test_vuln_det_all_tier_2.log 2021-08-09 Matias :yellow_circle:
test_vuln_det_all_tier_3.log 2021-08-09 Matias :red_circle: test_general_settings_ignore_time and test_providers_multiple_providers
mdengra commented 2 years ago

2021-08-17

I am trying to reproduce the random error produced by the test_validate_feed_content.py. I verify that in my testing environment the test does not fail when executed individually:

Used Wazuh-QA branch: 1531-full-yellow-vuln-det Test results with the default settings in the ossec.conf:

Test Executions Date By Status
test_validate_feed_content_local_r1.log 2021-08-17 Miguel :yellow_circle:
test_validate_feed_content_local_r2.log 2021-08-17 Miguel :yellow_circle:
test_validate_feed_content_local_r3.log 2021-08-17 Miguel :yellow_circle:
test_validate_feed_content_local_r4.log 2021-08-17 Miguel :yellow_circle:

After the previous results I have run the test_feeds group which is where the affected test is located:

Test Executions Date By Status Comment
test_feeds_local_r1.log 2021-08-17 Miguel :yellow_circle:
test_feeds_local_r2.log 2021-08-17 Miguel :red_circle: Error in test_extra_tags_canonical_feed (known failure [closed issue])
test_feeds_local_r3.log 2021-08-17 Miguel :yellow_circle:
test_feeds_local_r4.log 2021-08-17 Miguel :red_circle: Error in test_extra_tags_canonical_feed (known failure [closed issue])

Due to the errors detected in the test test_extra_tags_canonical_feed in which the associated issue is closed (#1545), I re-run the tests before reopening it:

Test Executions Date By Status Comment
test_feeds_local_r5.log 2021-08-17 Miguel :yellow_circle:
test_feeds_local_r6.log 2021-08-17 Miguel :red_circle: Error in test_extra_tags_canonical_feed (known failure [closed issue])
test_feeds_local_r7.log 2021-08-17 Miguel :yellow_circle:

I reopen #1545 due to the bug described in the issue: https://github.com/wazuh/wazuh/issues/9309

mdengra commented 2 years ago

2021-08-18

Full running

Used Wazuh-QA branch: 1531-full-yellow-vuln-det Test results with the default settings in the ossec.conf:

Test Executions Date By Status Comment
test_vulnerability_detector_local_r1.log 2021-08-18 Miguel :yellow_circle:
test_vulnerability_detector_local_r2.log 2021-08-18 Miguel :red_circle: Multiple errors in test_feeds/canonical/* (test cancelled after 8h of running time)
test_vulnerability_detector_local_r3.log 2021-08-18 Miguel :yellow_circle:

Results: The errors may be caused by the test_extra_tags_canonical_feed test of the issue https://github.com/wazuh/wazuh-qa/issues/1545. This test is the first one to run, and after the failure of this one, the following ones have continued to fail.

mdengra commented 2 years ago

2021-08-18

Full running

After disabling the test_extra_tags_canonical_feed test I re-run the full tests to verify the full yellow.

Used Wazuh-QA branch: 1545-disable-test-extra-tags-canonical-feed Test results with the default settings in the ossec.conf:

Test Executions Date By Status Comment
test_vulnerability_detector_local_r1.log 2021-08-18 Miguel :red_circle: Error in test_invalid_values_redhat_feed.py
test_vulnerability_detector_local_r2.log 2021-08-18 Miguel :yellow_circle:
test_vulnerability_detector_local_r3.log 2021-08-18 Miguel :yellow_circle:

Results: In the test_invalid_values_redhat_feed.py test, an error has been detected:

test_vulnerability_detector/test_feeds/redhat/test_invalid_values_redhat_feed.py::test_invalid_redhat_feed[REDHAT_configuration-tag: tests, value: ] FAILED

I will run this test in isolation to see if the above error is repeatable:

Test Executions Date By Status
test_invalid_values_redhat_feed_local_r1.log 2021-08-18 Miguel :yellow_circle:
test_invalid_values_redhat_feed_local_r2.log 2021-08-18 Miguel :yellow_circle:
test_invalid_values_redhat_feed_local_r3.log 2021-08-18 Miguel :yellow_circle:

Results: After three runs, the error has not been repeated, so the cause of the error should be further researched.

damarisg commented 2 years ago

Full Execution

Test Executions Date By Status
FullYellowVuln.log 2021-08-18 Seyla :yellow_circle:
FullYellowVuln3.log 2021-08-18 Seyla :yellow_circle:
FullYellowVuln2.log 2021-08-18 Seyla :yellow_circle:
snaow commented 2 years ago

Closed by https://github.com/wazuh/wazuh-qa/pull/1755

Good job.