Generate isolated resources for each AWS integration test

[GGP1]

Description

During the investigation in https://github.com/wazuh/wazuh/issues/18478, we've found that the AWS integration tests create resources using hardcoded parameters, meaning that every test tries to create the same resource and they fail if it already exists. Because of this, it's not possible to run two suites in paralell.

In order to solve this, we should generate isolated and temporary resources with randomly created parameters (account_id, name, etc.) to avoid conflicts. We could include a cleanup function that deletes the resources created after the execution.

Current behavior

The tests fail because of already existing resources.

Expected behavior

The tests generate isolated and temporary resources for each execution.

[EduLeon12]

Issue Update

I have set up a testing environment that consists of 2 vagrant machines with all the necessary tools to run the AWS ITs, all tests will be run simultaneously to determine exactly which test and more specifically which resource will cause an error on parallel executions

First VM details

```command root@vagrant:/wazuh/tests/integration# cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.3 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.3 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy root@vagrant:/wazuh/tests/integration# uname -a Linux vagrant 5.15.0-83-generic #92-Ubuntu SMP Mon Aug 14 09:30:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ``` VM ID ```command root@vagrant:/wazuh/tests/integration/test_aws# dmidecode -s system-uuid fbc1928d-7955-9b42-89b7-9717dfe5a251 ```

Second VM details

```command root@vagrant:/wazuh/tests/integration/test_aws# cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.3 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.3 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy root@vagrant:/wazuh/tests/integration/test_aws# uname -a Linux vagrant 5.15.0-83-generic #92-Ubuntu SMP Mon Aug 14 09:30:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ``` VM ID ```command root@vagrant:/wazuh/tests/integration/test_aws# dmidecode -s system-uuid 4b9b295c-828a-644f-b0bd-bfbddc4c82c4 ```

Testing Results

Test name	Paralel Execution
test_aws/test_basic.py	🟢
test_aws/test_custom_bucket.py	🟢
test_aws/test_discard_regex.py	🟢
test_aws/test_log_groups.py	🔴
test_aws/test_only_logs_after.py	🔴
test_aws/test_parser.py	🟢
test_aws/test_path.py	🔴
test_aws/test_path_suffix.py	🔴
test_aws/test_regions.py	🔴
test_aws/test_remove_from_bucket.py	🟢

Test logs

test_basic 🟢

```python root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_basic.py =================================== test session starts =================================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 16 items test_basic.py ................ [100%] ======================= 16 passed, 7 warnings in 336.66s (0:05:36) ======================== root@vagrant:/wazuh/tests/integration/test_aws# ────────────────────────────────────────────────────────────────────────────────────────── root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_basic.py =================================== test session starts =================================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 16 items test_basic.py ................ [100%] ======================= 16 passed, 5 warnings in 329.51s (0:05:29) ======================== root@vagrant:/wazuh/tests/integration/test_aws# ```

test_custom_bucket 🟢

```python root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_custom_bucket.py =================================== test session starts =================================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 2 items test_custom_bucket.py .. [100%] ────────────────────────────────────────────────────────────────────────────────────────── root@vagrant:/wazuh/tests/integration/test_aws# pytest test_custom_bucket.py ========================= test session starts ========================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 2 items test_custom_bucket.py .. [100%] ==================== 2 passed, 5 warnings in 43.81s ==================== ```

test_discard_regex 🟢

```python root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_discard_regex.py ===================================================================================== test session starts ====================================================================================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 17 items test_discard_regex.py ................. [100%] ========================================================================== 17 passed, 7 warnings in 280.54s (0:04:40) ========================================================================== ────────────────────────────────────────────────────────────────────────────────────────── root@vagrant:/wazuh/tests/integration/test_aws# pytest test_discard_regex.py =============================== test session starts =============================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 17 items test_discard_regex.py ................. [100%] ============== 17 passed, 5 warnings in 278.25s (0:04:38) ============== =========================================================================================== FAILURES =========================================================================================== ```

test_log_groups 🔴

```python root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_log_groups.py ===================================================================================== test session starts ====================================================================================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 2 items test_log_groups.py .. [100%] ================================================================================ 2 passed, 7 warnings in 47.69s ================================================================================ ────────────────────────────────────────────────────────────────────────────────────────── root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_log_groups.py ===================================================================================== test session starts ====================================================================================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 2 items test_log_groups.py EF [100%] ============================================================================================ ERRORS ============================================================================================ ```

test_only_logs_after 🔴

```python root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_only_logs_after.py ===================================================================================== test session starts ====================================================================================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 47 items test_only_logs_after.py F..FF.....F.F.F..............................x. [100%] ────────────────────────────────────────────────────────────────────────────────────────── root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_only_logs_after.py ===================================================================================== test session starts ====================================================================================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 47 items test_only_logs_after.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFxFE ```

test_parser 🟢

```python root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_parser.py ===================================================================================== test session starts ====================================================================================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 26 items test_parser.py ..............F........... [100%] ────────────────────────────────────────────────────────────────────────────────────────── root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_parser.py ===================================================================================== test session starts ====================================================================================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 26 items test_parser.py ..............F........... [100%] ```

test_path 🔴

```python root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_path.py ===================================================================================== test session starts ====================================================================================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 42 items test_path.py .......................................... [100%] ========================================================================== 42 passed, 7 warnings in 923.10s (0:15:23) ========================================================================== ────────────────────────────────────────────────────────────────────────────────────────── root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_path.py ===================================================================================== test session starts ====================================================================================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 42 items test_path.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF [100%] =========================================================================================== FAILURES =========================================================================================== ```

test_path_suffix 🔴

```python root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_path_suffix.py ===================================================================================== test session starts ====================================================================================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 9 items test_path_suffix.py ......... [100%] ========================================================================== 9 passed, 7 warnings in 195.22s (0:03:15) =========================================================================== ────────────────────────────────────────────────────────────────────────────────────────── root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_path_suffix.py ===================================================================================== test session starts ====================================================================================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 9 items test_path_suffix.py FFFFFFFFF [100%] =========================================================================================== FAILURES =========================================================================================== ```

test_regions 🔴

```python root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_regions.py ===================================================================================== test session starts ====================================================================================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 24 items test_regions.py ........................ [100%] ========================================================================== 24 passed, 8 warnings in 555.34s (0:09:15) ========================================================================== ────────────────────────────────────────────────────────────────────────────────────────── root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_regions.py ===================================================================================== test session starts ====================================================================================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 24 items test_regions.py FFFFFFFFFFFFFFFFFFFFFFFF [100%] =========================================================================================== FAILURES =========================================================================================== ```

test_remove_from_bucket 🔴

```python root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_remove_from_bucket.py ===================================================================================== test session starts ====================================================================================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: metadata-3.0.0, html-3.1.1 collected 15 items test_remove_from_bucket.py ...sss......... [100%] ==================================================================== 12 passed, 3 skipped, 7 warnings in 287.86s (0:04:47) ===================================================================== ────────────────────────────────────────────────────────────────────────────────────────── root@ubuntu2204:/wazuh/tests/integration/test_aws# pytest test_remove_from_bucket.py ============================== test session starts =============================== platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0 rootdir: /wazuh/tests/integration, configfile: pytest.ini plugins: html-3.1.1, metadata-3.0.0 collected 15 items test_remove_from_bucket.py ...sss......... [100%] ```

[EduLeon12]

Issue update

During the testing and debugging of the test_log_groups and the test_remove_from_bucket, it was found that to test a successful parallel execution, a log stream was tried to be created resulting in an error due to lack of permission to create it.

An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: arn:aws:iam::966237403726:user/eduardo.leon is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:us-east-1:966237403726:log-group:temporary-log-group-9935:log-stream: because no identity-based policy allows the logs:CreateLogGroup action

The permission has already been requested and the test will continue once it has been granted.

[EduLeon12]

Issue Update

After discussing with the team manager and analyzing different approaches to the issue it was decided to add new fixtures to isolate the creation of every needed AWS resource for the launch of the tests, these will include S3 buckets and services with an added hash to ensure entropy within the parallel executions of tests.

The official AWS documentation regarding the cost of resources states, that there is no cost associated with the creation of the buckets and log groups or streams, and since the size of logs used for every test averages 50B, it can be concluded that there is not going to be an increase of cost on the execution of tests.

This will also allow us to not depend on a specific AWS account to run the tests but be able to run them in any sandbox account.

Currently working on the S3 fixture extracting the parameters from the data set of the tests and applying them to the creation of the bucket endpoint, this is being added to the framework alongside other methods of AWS resources manipulation.

[EduLeon12]

Issue Update

After discussing with the team about possible implementations for the fixture they will be made following these guidelines:

• A session fixture to create a session uuid for every resource to allow parallel executions.
• A function fixture to add the uuid to the bucket name retrieved from the data.
• A session fixture to include and track all resources created to later delete them.
• Modify the services fixtures to follow the same logic.

All fixtures have been developed. The next step is to test the fixtures and ensure tests are passing with the inclusion of the uuid on the name.

[EduLeon12]

Issue Update

After testing the changes it was found that there were some flaws in the design developed, Some due to errors in the scopes of the fixtures, and also it wasn't as readable.

For better design, the qa-integration-framework was reworked to include all boto methods for the creation of the resources and remove the constant such as the permanent log groups as the default parameter for the existing creation methods. Also, the methods were changed for better modularization and to abstract the logic of every resource to its method and an error handling was added for each method. framework branch 4714-generate-isolated-resources

On the other hand, The fixtures of the core repository were also reworked, and each resource method now includes only its creation and later addition to a new list of all resources for them to be deleted upon completion of the test.

Finally, all metadata has been edited to include a new key called resource_type to identify the type of resource and be able to delete afterward by calling the corresponding method.

[EduLeon12]

Issue Update

Working on core repository fixtures import errors due to the changes. Some fixtures are under a big refactor since the deletion part of the resource it's only necessary for the log group and the buckets, because all resources created inside these will be deleted as well.

Because of the mentioned behaviour the logic of yielding filenames, log streams or events is no longer needed. Therefore, it is being removed and optimized.

[EduLeon12]

Issue Update

After a back and forth on the modification of the qa-integration-framework and the design all fixtures have been modified. All are ready to be tested. Test basic data has been modified to include new metadata parameters to test the fixture calls once is concluded that it's passing without errors, the rest of the tests will be change accordingly and tested as well.

[EduLeon12]

Issue Update

Working on fixing various import and execution errors, adapting imports, and standardizing boto interface to be client interface only.

[EduLeon12]

Issue Update

After several debugs of boto errors, It was found that the errors were due to versioning, Framework outdated version of boto has different parameters on the API calls for resource creation. Some tests will be launched to determine if upgrading the package in the testing framework is not a breaking change.

[EduLeon12]

Issue update

All testing on the creation and deletion of resources has been tested successfully. However, it was found that a new fixture to edit the test cases was needed.

The current test works by loading into the ossec.conf a configuration based on the content of the data file to trigger the wodle, but since the fixture in charge of creating the ossec.conf is not changing the config yaml file but only the test metadata is not impacting on the wodle configuration resulting in the test to fail. To fix this a new fixture is being developed to follow the same behavior of the framework (ossec.conf) configuration file, its intended behavior will be to:

• Load and backup current data file.
• Allow the creation of the resource fixture to overwrite the data file.
• Restore the data file once the test is completed.

[EduLeon12]

Issue Update

After testing the proposal, It was changed. The configurator is now in charge of managing the resources and configuring the files accordingly. Configurator is currently under work to be extended to be able to perform the following tasks:

• [x] Create a backup of the cases found in the cases YAML file.
• [x] Modify the cases file to include a new test session ID (Now created by the configurator).
• [x] Track all resources being modified using a set to avoid creating repeated resources.
• [x] Create all testing resources.
• [x] Delete all created testing resources.
• [x] Restore cases YAML file using the backup.

The creation and deletion still require some though and work process since the resource_type is needed to verify the type of resource that is going to be created and later on deleted.

[EduLeon12]

Issue Update

It was found that the fixtures needed modification because the encapsulation of the configurator had to be changed to a test wrapper to make it dynamic and usable as a fixture for each test in every module. This change allows them to use the same session_id for resource identification during parallel runs.

[EduLeon12]

Issue update

It was decided to also add a decorator to set the module name for the configurator for each test. The decorator will only be used in the first test of every module to avoid repeating it on each test and only needing to add what is necessary in each configure_test decorator for each test.

[EduLeon12]

Issue update

After realizing an error during the execution of the design due to the incompatibility of the timing imports between decorators and fixtures it was decided to go with a more modular design. Abstracting the responsibility of the configurator according to what was said here.

It was consulted with the manager to conclude the design regarding the responsibility of the creation of the resources. Either leave it to the configurator or the fixtures and allow a more visible approach to each step of the test.

[EduLeon12]

Issue Update

• [x] Completion of development of fixtures and implementation on test modules.
• [x] Tested the resource creation and closure handler.

Currently debugging an error in the configuration setup of the module, which is not setting the profile used making the regex not match causing the test to fail.

[EduLeon12]

Issue update

• Test basic suite has been launched successfully, and all resources used during test were created and deleted as expected.

root@vagrant:/wazuh/tests/integration/test_aws# pytest -sx --disable-warnings test_basic.py
================================ test session starts ================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 16 items                                                          

test_basic.py ................

==================== 16 passed, 2 warnings in 294.42s (0:04:54) =====================

Currently working on the custom bucket test suite on the upload_file fixture and boto method. All the boto methods to create and modify the sqs queue for the bucket have been developed and tested successfully through their corresponding developed fixture.

[EduLeon12]

Issue update

The upload of the file has been developed and tested. Also, a new fixture and boot method to delete all objects found inside the bucket had to be developed to allow the deletion of the bucket due to a rule of AWS that doesn't allow deletion of non empty buckets.

• Custom bucket test suite has been successfully run. A time sleep had to be inserted to allow the queue to receive the message for the regex to be able to match it.

root@vagrant:/wazuh/tests/integration/test_aws# pytest -sx --disable-warnings test_custom_bucket.py
======================== test session starts =========================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collecting ... 
collected 2 items

test_custom_bucket.py ..

============== 2 passed, 2 warnings in 75.93s (0:01:15) ==============

[EduLeon12]

Issue Update

Due to internal changes, the tasks needed to complete this issue based on each test module will be listed.

[!NOTE] This tasks are not conclusive, they are just a proposal. They might require redesigns, changes, or more work to achieve.

For all modules.

• [ ] Add the resource_type tag and value to each data module.
• [ ] Remove the profile for the configuration template of the module since the profile will be handled via ENV.
• [ ] Add a logic block In the fixture manage_bucket_files to manage the amount of files that will be created.
  • This logic should take into consideration the creation of multiple files.
  • If all tests are creating the file to the same bucket no change needs to be added since they will be deleted on the cleanup phase of the test.
• [ ] Test the creation of the log stream and logs inside a recently created log group.
• [ ] Add the necessary fixtures to each test.

Discard regex

- [ ] Ensure the discard_regex configuration is being inserted as expected (This wasn't changed and should not require a change). - [ ] Test the regex parameter and check the log monitor response to ensure the assertion is the expected.

Log groups

- [ ] Apply the same logic of the `manage_bucket_files` but for the messages of the log-stream.

Only logs after

- [ ] If the logic of the creation of the file was handled successfully this module should not require any changes.

Parser

- [ ] Since the metadata is missing from the data, Key errors will arise before the Wazuh setup phase. This behavior needs to be handled accordingly. Either capture the exceptions to allow the continuation of the test to reach to the point where the logs will be handled to assert the tests or change the logic regarding the tests. If it is decided to manage the exception the use of a logger

Path Suffix

- [ ] Manage the **key** value passed to the file creation to ensure the `path` is being created as expected.

Path

- [ ] Should be the same logic as above

Remove from bucket

- [ ] The `file exist` method was deleted from the current flow since my thought was to delete the interaction of the tests with the testing framework, I think that responsibility belongs to the fixture so a redesign of the block will need to be made if it is concluded to be that way. Ex of change: ```python # From assert not file_exists(filename=metadata['uploaded_file'], bucket_name=bucket_name) # To expected_log = "INSERT LOG WHEN THE BUCKET OR LOG STREAM IS EMPTY" log_monitor.start( timeout=session_parameters.default_timeout, callback=event_monitor.make_aws_callback(expected_log) ) ``` Also, if multiple files are present in the bucket when asserting and the empty bucket is not a reassurance that the removal was correct the regex can be defined to look for the file using the specific key assigned to the metadata of the test and assert for that the callback_result should be None since no match should occur.

Regions

- [ ] This module is the one that will require the biggest change or thought process, since the current design has the region set as an environment variable and this module needs to create resources in multiple regions. Multiple solutions can be performed for this. - [ ] Create a new fixture for the creation of the resource of another region that will be received by the list of regions in the metadata of the test. - [ ] Modify the resource creation fixture to have the region as a parameter instead based on the content of the metadata and create the resource in said region, this will also require a modification in the deletion part. - [ ] Create a session of BOTO for every region or extra region set in the fixtures and manage the creation in a local scope.

[fdalmaup]

Conclusion

Some minor changes were left, but since the issue points to an epic with a specific branch, the team considers that these can be carried out in https://github.com/wazuh/wazuh/issues/22514 and https://github.com/wazuh/qa-integration-framework/issues/133 which will also point to the epic to avoid overextending this issue.