Refactor Azure module to use Logger tool and log relevant information

EduLeon12 commented 1 year ago

Affected integration
Azure

Description

As described in #16717, To standardize the logging a refactor in this cloud module needs to be done to integrate the logging tool developed in the mentioned issue.

Logging refactor on the module should be done to log relevant information about the process.

Unit tests should be reviewed as well to adapt to the new logging method.

Tasks

[x] Test in a manager.
[x] Test in an agent.
[x] Unit tests without failures. Updated if there are any relevant changes.
[x] Integration tests without failures. Updated if there are any relevant changes.
[x] Update the documentation if necessary.
[x] Add entry to the changelog if necessary.

EduLeon12 commented 1 year ago

Logger implemented following the current behavior. It was found it follows the same issue as the aws_logger, meaning it depends on developing a parser to log the output.

A discussion was opened to confirm how to proceed.

EduLeon12 commented 1 year ago

Issue Update

Logger implemented and parser tested. Test messages can be seen in the ossec.log

2023/10/04 17:34:40 wazuh-modulesd:azure-logs[9536] wm_parser.c:76 at wm_parse_output(): INFO: This is an INFO test Message
2023/10/04 17:34:40 wazuh-modulesd:azure-logs[9536] wm_parser.c:82 at wm_parse_output(): WARNING: This is a WARNING test Message
2023/10/04 17:34:40 wazuh-modulesd:azure-logs[9536] wm_parser.c:80 at wm_parse_output(): ERROR: This is an ERROR test Message

After running the unit test for the Azure wodle the results are very negative but expected.

============================ 54 failed, 94 passed, 55 warnings in 4.18s ============================

Next steps:

[x] Fix unit test.
[x] Do a full testing in agents for python version (3.9 and 3.7)

EduLeon12 commented 1 year ago

Issue Update

All changes to the module have been made according to the implemented logger and parser.

Testing

Manager logs

The testing was made by adding a testing message for every level with the `debug` level set to 2. ``` root@wazuh-master:/var/ossec# cat etc/local_internal_options.conf wazuh_modules.debug=2 ``` - `ossec.log` ``` 2023/10/05 19:28:42 wazuh-modulesd:azure-logs[5927] wm_parser.c:76 at wm_parse_output(): INFO: This is an INFO test Message 2023/10/05 19:28:42 wazuh-modulesd:azure-logs[5927] wm_parser.c:82 at wm_parse_output(): WARNING: This is a WARNING test Message 2023/10/05 19:28:42 wazuh-modulesd:azure-logs[5927] wm_parser.c:80 at wm_parse_output(): ERROR: This is an ERROR test Message 2023/10/05 19:28:42 wazuh-modulesd:azure-logs[5927] wm_parser.c:80 at wm_parse_output(): ERROR: This is an ERROR test Message, with more info ``` And the module was functioning properly for the azure storage integration. ``` 2023/10/05 19:28:42 wazuh-modulesd:azure-logs[5927] wm_parser.c:76 at wm_parse_output(): INFO: Azure Storage starting. 2023/10/05 19:28:42 wazuh-modulesd:azure-logs[5927] wm_parser.c:76 at wm_parse_output(): INFO: Storage: Authenticating. 2023/10/05 19:28:42 wazuh-modulesd:azure-logs[5927] wm_parser.c:76 at wm_parse_output(): INFO: Storage: Authenticated. 2023/10/05 19:28:42 wazuh-modulesd:azure-logs[5927] wm_parser.c:76 at wm_parse_output(): INFO: Storage: Getting blobs. 2023/10/05 19:28:42 wazuh-modulesd:azure-logs[5927] wm_parser.c:76 at wm_parse_output(): INFO: Storage: The search starts from the date: 2023-10-04 19:28:42.218033+00:00 for blobs in container: 'null-json-blob' and prefix: '/' 2023/10/05 19:28:42 wazuh-modulesd:azure-logs[5927] wm_parser.c:76 at wm_parse_output(): INFO: Storage: End 2023/10/05 19:28:42 wazuh-modulesd:azure-logs[5927] wm_azure.c:369 at wm_azure_storage(): INFO: Finished Storage log collection for container 'null-json-blob'. 2023/10/05 19:28:42 wazuh-modulesd:azure-logs[5927] wm_azure.c:94 at wm_azure_main(): INFO: Finished Storage log collection for 'storage_53336128'. 2023/10/05 19:28:42 wazuh-modulesd:azure-logs[5927] wm_azure.c:100 at wm_azure_main(): DEBUG: Fetching logs finished. 2023/10/05 19:28:42 wazuh-modulesd:azure-logs[5927] wm_azure.c:70 at wm_azure_main(): DEBUG: Sleeping until: 2023/10/06 19:28:41 ```

Azure unit test

``` (unit-test) eduardoleon@pop-os:~/git/wazuh/wodles(16739-refactor-azure-logs)$ pytest --disable-warnings azure/tests/ ============================== test session starts ============================== platform linux -- Python 3.9.16, pytest-7.4.2, pluggy-0.13.1 rootdir: /home/eduardoleon/git/wazuh/wodles plugins: anyio-3.6.2, aiohttp-1.0.4, trio-0.7.0, html-3.0.0, asyncio-0.18.1, cov-3.0.0, metadata-2.0.2 asyncio: mode=auto collected 144 items azure/tests/test_azure.py ............................................... [ 32%] .................................................................. [ 78%] azure/tests/test_orm.py ............................... [100%] ======================== 144 passed, 1 warning in 0.44s ========================= ``` - Coverage Report ``` (unit-test) eduardoleon@pop-os:~/git/wazuh(16738-refactor-gcp-logs)$ coverage run --omit=test -m pytest --disable-warnings wodles/azure/tests && coverage report -m =============================== test session starts ================================ platform linux -- Python 3.9.16, pytest-7.4.2, pluggy-0.13.1 rootdir: /home/eduardoleon/git/wazuh plugins: anyio-3.6.2, aiohttp-1.0.4, trio-0.7.0, html-3.0.0, asyncio-0.18.1, cov-3.0.0, metadata-2.0.2 asyncio: mode=auto collected 148 items wodles/azure/tests/test_azure.py ........................................... [ 29%] .......................................................................... [ 79%] wodles/azure/tests/test_orm.py ............................... [100%] ========================== 148 passed, 1 warning in 0.56s ========================== /home/eduardoleon/git/wazuh/wodles/azure/tests/test_azure.py:744: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(blob.properties.content_length.call_count, 1) /home/eduardoleon/git/wazuh/wodles/azure/tests/test_azure.py:746: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(blob_service.get_blob_to_text.call_count, 0) Name Stmts Miss Cover Missing ---------------------------------------------------------------- wodles/azure/azure-logs.py 447 16 96% 710, 713, 722, 836, 884-898 wodles/azure/orm.py 141 0 100% wodles/azure/tests/test_azure.py 571 1 99% 53 wodles/azure/tests/test_orm.py 149 0 100% wodles/utils.py 57 27 53% 29-30, 39-40, 59-66, 84-98, 111, 124, 137 ---------------------------------------------------------------- TOTAL 1365 44 97% ```

Only test remains are testing the other Azure services in Agents with different Python versions 3.9 and 3.7.

EduLeon12 commented 1 year ago

Issue Update

Currently trying to fix an issue where the parser is not printing the logs in the ossec.log when executed on agents. But, when testing the module manually it works without problems.

root@1048e2f799e9:/var/ossec/wodles/azure# ./azure-logs --storage --storage_auth_path /var/ossec/etc/azure_credentials --container null-json-blob --blobs .json --storage_tag storage_248988139 --json_inline --storage_time_offset 24h --debug 2
:azure_wodle: - INFO - Azure Storage starting.
INFO::azure_wodle::Azure Storage starting.
:azure_wodle: - INFO - This is an INFO test Message
INFO::azure_wodle::This is an INFO test Message
:azure_wodle: - WARNING - This is a WARNING test Message
WARNING::azure_wodle::This is a WARNING test Message
:azure_wodle: - ERROR - This is an ERROR test Message
ERROR::azure_wodle::This is an ERROR test Message
:azure_wodle: - ERROR - This is an ERROR test Message, with more info
NoneType: None
ERROR::azure_wodle::This is an ERROR test Message, with more info
NoneType: None
:azure_wodle: - INFO - Storage: Authenticating.
INFO::azure_wodle::Storage: Authenticating.
:azure_wodle: - INFO - Storage: Authenticated.
INFO::azure_wodle::Storage: Authenticated.
:azure_wodle: - INFO - Storage: Getting blobs.
INFO::azure_wodle::Storage: Getting blobs.
:azure_wodle: - INFO - Storage: The search starts from the date: 2023-10-05 19:59:37.422698+00:00 for blobs in container: 'null-json-blob' and prefix: '/'
INFO::azure_wodle::Storage: The search starts from the date: 2023-10-05 19:59:37.422698+00:00 for blobs in container: 'null-json-blob' and prefix: '/'
:azure_wodle: - INFO - Storage: End
INFO::azure_wodle::Storage: End

Also, trying to fix an installation error when trying to install python 3.7 to test on agent.

EduLeon12 commented 10 months ago

Issue update

After rebasing the branch to contain the latest changes the test message can be seen in the ossec.log using the wm_parse_output method as expected. Concluding that the module is working as expected as well as the modulesd parsing of the logs.

2023/12/20 17:11:05 wazuh-modulesd:azure-logs[47159] wm_azure.c:356 at wm_azure_storage(): DEBUG: Launching command: wodles/azure/azure-logs --storage --storage_auth_path /var/ossec/wodles/wodles_credentials/azure_storage_credentials --container "frameworktestcontainer" --blobs "*" --storage_tag azure-activity --storage_time_offset 30d --debug 2
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:76 at wm_parse_output(): INFO: Azure Storage starting.
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:76 at wm_parse_output(): INFO: TEST MESSAGE: INFO
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:76 at wm_parse_output(): DEBUG: TEST MESSAGE: DEBUG
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:82 at wm_parse_output(): WARNING: TEST MESSAGE: WARNING
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:80 at wm_parse_output(): ERROR: TEST MESSAGE: ERROR
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:76 at wm_parse_output(): INFO: Storage: Authenticating.
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:76 at wm_parse_output(): INFO: Storage: Authenticated.
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:76 at wm_parse_output(): INFO: Storage: Getting blobs.
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:76 at wm_parse_output(): INFO: Storage: The search starts from the date: 2023-11-20 17:11:08.081019+00:00 for blobs in container: 'frameworktestcontainer' and prefix: '/'
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:76 at wm_parse_output(): INFO: Storage: End
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_azure.c:369 at wm_azure_storage(): INFO: Finished Storage log collection for container 'frameworktestcontainer'.
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_azure.c:94 at wm_azure_main(): INFO: Finished Storage log collection for 'azure-activity'.
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_azure.c:100 at wm_azure_main(): DEBUG: Fetching logs finished.
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_azure.c:70 at wm_azure_main(): DEBUG: Sleeping until: 2023/12/20 17:21:05

Agent output is also as expected.

2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_parser.c:76 at wm_parse_output(): INFO: Azure Storage starting.
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_parser.c:76 at wm_parse_output(): INFO: Test_message: INFO
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_parser.c:76 at wm_parse_output(): DEBUG: Test_message: DEBUG
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_parser.c:82 at wm_parse_output(): WARNING: Test_message: WARNING
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_parser.c:80 at wm_parse_output(): ERROR: Test_message: ERROR
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_parser.c:76 at wm_parse_output(): INFO: Storage: Authenticating.
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_parser.c:80 at wm_parse_output(): ERROR: Error: The authentication file could not be opened: [Errno 2] No such file or directory: '/var/ossec/wodles_credentials/azure_storage_credentials'
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_azure.c:369 at wm_azure_storage(): INFO: Finished Storage log collection for container 'frameworktestcontainer'.
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_azure.c:94 at wm_azure_main(): INFO: Finished Storage log collection for 'azure-activity'.
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_azure.c:100 at wm_azure_main(): DEBUG: Fetching logs finished.
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_azure.c:70 at wm_azure_main(): DEBUG: Sleeping until: 2023/12/20 18:29:19

Finally unit test were relaunched to ensure wodles its working,

(unit-test) eduardoleon@pop-os:~/git/wazuh(16739-refactor-azure-logs)$ pytest wodles/
=========================== test session starts ===========================
platform linux -- Python 3.9.16, pytest-7.4.2, pluggy-0.13.1
rootdir: /home/eduardoleon/git/wazuh
plugins: anyio-3.6.2, aiohttp-1.0.4, trio-0.7.0, html-3.0.0, asyncio-0.18.1, cov-3.0.0, metadata-2.0.2
asyncio: mode=auto
collected 852 items

wodles/aws/tests/test_aws_bucket.py ............................... [  3%]
................................................................... [ 11%]
................................................................... [ 19%]
............................................                        [ 24%]
wodles/aws/tests/test_aws_s3.py ..................                  [ 26%]
wodles/aws/tests/test_aws_service.py ....                           [ 27%]
wodles/aws/tests/test_cloudtrail.py ..                              [ 27%]
wodles/aws/tests/test_cloudwatchlogs.py ........................... [ 30%]
..........................                                          [ 33%]
wodles/aws/tests/test_config.py ................................... [ 37%]
...........................................                         [ 42%]
wodles/aws/tests/test_guardduty.py .................                [ 44%]
wodles/aws/tests/test_inspector.py ......                           [ 45%]
wodles/aws/tests/test_load_balancers.py ............                [ 46%]
wodles/aws/tests/test_s3_log_handler.py ....                        [ 47%]
wodles/aws/tests/test_server_access.py ............................ [ 50%]
.....                                                               [ 51%]
wodles/aws/tests/test_sqs_queue.py .......                          [ 51%]
wodles/aws/tests/test_tools.py ...............................      [ 55%]
wodles/aws/tests/test_umbrella.py ......                            [ 56%]
wodles/aws/tests/test_vpcflow.py ............................       [ 59%]
wodles/aws/tests/test_waf.py .......                                [ 60%]
wodles/aws/tests/test_wazuh_integration.py ........................ [ 63%]
................................................................... [ 71%]
...........                                                         [ 72%]
wodles/azure/tests/test_azure.py .................................. [ 76%]
................................................................... [ 84%]
............                                                        [ 85%]
wodles/azure/tests/test_orm.py ...............................      [ 89%]
wodles/docker-listener/tests/test_docker_listener.py .............. [ 90%]
.....                                                               [ 91%]
wodles/gcloud/tests/test_bucket.py ................................ [ 95%]
.                                                                   [ 95%]
wodles/gcloud/tests/test_gcloud.py .........                        [ 96%]
wodles/gcloud/tests/test_integration.py ........                    [ 97%]
wodles/gcloud/tests/test_subscriber.py ..............               [ 99%]
wodles/gcloud/tests/test_tools.py ........                          [100%]

============================ warnings summary =============================
../../.pyenv/versions/3.9.16/envs/unit-test/lib/python3.9/site-packages/pytest_aiohttp/plugin.py:28
  /home/eduardoleon/.pyenv/versions/3.9.16/envs/unit-test/lib/python3.9/site-packages/pytest_aiohttp/plugin.py:28: DeprecationWarning: The 'asyncio_mode' is 'legacy', switching to 'auto' for the sake of pytest-aiohttp backward compatibility. Please explicitly use 'asyncio_mode=strict' or 'asyncio_mode=auto' in pytest configuration file.
    config.issue_config_time_warning(LEGACY_MODE, stacklevel=2)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================== 852 passed, 1 warning in 3.08s ======================

Conclusion

The logger has been successfully integrated and the module is logging the expected messages.

wazuh / wazuh