Closed EduLeon12 closed 10 months ago
Logger implemented following the current behavior. It was found it follows the same issue as the aws_logger, meaning it depends on developing a parser to log the output.
A discussion was opened to confirm how to proceed.
Logger implemented and parser tested. Test messages can be seen in the ossec.log
2023/10/04 17:34:40 wazuh-modulesd:azure-logs[9536] wm_parser.c:76 at wm_parse_output(): INFO: This is an INFO test Message
2023/10/04 17:34:40 wazuh-modulesd:azure-logs[9536] wm_parser.c:82 at wm_parse_output(): WARNING: This is a WARNING test Message
2023/10/04 17:34:40 wazuh-modulesd:azure-logs[9536] wm_parser.c:80 at wm_parse_output(): ERROR: This is an ERROR test Message
After running the unit test for the Azure wodle the results are very negative but expected.
============================ 54 failed, 94 passed, 55 warnings in 4.18s ============================
Next steps:
All changes to the module have been made according to the implemented logger and parser.
Only test remains are testing the other Azure services in Agents with different Python versions 3.9 and 3.7.
Currently trying to fix an issue where the parser is not printing the logs in the ossec.log
when executed on agents. But, when testing the module manually it works without problems.
root@1048e2f799e9:/var/ossec/wodles/azure# ./azure-logs --storage --storage_auth_path /var/ossec/etc/azure_credentials --container null-json-blob --blobs .json --storage_tag storage_248988139 --json_inline --storage_time_offset 24h --debug 2
:azure_wodle: - INFO - Azure Storage starting.
INFO::azure_wodle::Azure Storage starting.
:azure_wodle: - INFO - This is an INFO test Message
INFO::azure_wodle::This is an INFO test Message
:azure_wodle: - WARNING - This is a WARNING test Message
WARNING::azure_wodle::This is a WARNING test Message
:azure_wodle: - ERROR - This is an ERROR test Message
ERROR::azure_wodle::This is an ERROR test Message
:azure_wodle: - ERROR - This is an ERROR test Message, with more info
NoneType: None
ERROR::azure_wodle::This is an ERROR test Message, with more info
NoneType: None
:azure_wodle: - INFO - Storage: Authenticating.
INFO::azure_wodle::Storage: Authenticating.
:azure_wodle: - INFO - Storage: Authenticated.
INFO::azure_wodle::Storage: Authenticated.
:azure_wodle: - INFO - Storage: Getting blobs.
INFO::azure_wodle::Storage: Getting blobs.
:azure_wodle: - INFO - Storage: The search starts from the date: 2023-10-05 19:59:37.422698+00:00 for blobs in container: 'null-json-blob' and prefix: '/'
INFO::azure_wodle::Storage: The search starts from the date: 2023-10-05 19:59:37.422698+00:00 for blobs in container: 'null-json-blob' and prefix: '/'
:azure_wodle: - INFO - Storage: End
INFO::azure_wodle::Storage: End
Also, trying to fix an installation error when trying to install python 3.7 to test on agent.
After rebasing the branch to contain the latest changes the test message can be seen in the ossec.log using the wm_parse_output method as expected. Concluding that the module is working as expected as well as the modulesd parsing of the logs.
2023/12/20 17:11:05 wazuh-modulesd:azure-logs[47159] wm_azure.c:356 at wm_azure_storage(): DEBUG: Launching command: wodles/azure/azure-logs --storage --storage_auth_path /var/ossec/wodles/wodles_credentials/azure_storage_credentials --container "frameworktestcontainer" --blobs "*" --storage_tag azure-activity --storage_time_offset 30d --debug 2
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:76 at wm_parse_output(): INFO: Azure Storage starting.
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:76 at wm_parse_output(): INFO: TEST MESSAGE: INFO
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:76 at wm_parse_output(): DEBUG: TEST MESSAGE: DEBUG
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:82 at wm_parse_output(): WARNING: TEST MESSAGE: WARNING
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:80 at wm_parse_output(): ERROR: TEST MESSAGE: ERROR
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:76 at wm_parse_output(): INFO: Storage: Authenticating.
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:76 at wm_parse_output(): INFO: Storage: Authenticated.
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:76 at wm_parse_output(): INFO: Storage: Getting blobs.
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:76 at wm_parse_output(): INFO: Storage: The search starts from the date: 2023-11-20 17:11:08.081019+00:00 for blobs in container: 'frameworktestcontainer' and prefix: '/'
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_parser.c:76 at wm_parse_output(): INFO: Storage: End
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_azure.c:369 at wm_azure_storage(): INFO: Finished Storage log collection for container 'frameworktestcontainer'.
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_azure.c:94 at wm_azure_main(): INFO: Finished Storage log collection for 'azure-activity'.
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_azure.c:100 at wm_azure_main(): DEBUG: Fetching logs finished.
2023/12/20 17:11:08 wazuh-modulesd:azure-logs[47159] wm_azure.c:70 at wm_azure_main(): DEBUG: Sleeping until: 2023/12/20 17:21:05
Agent output is also as expected.
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_parser.c:76 at wm_parse_output(): INFO: Azure Storage starting.
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_parser.c:76 at wm_parse_output(): INFO: Test_message: INFO
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_parser.c:76 at wm_parse_output(): DEBUG: Test_message: DEBUG
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_parser.c:82 at wm_parse_output(): WARNING: Test_message: WARNING
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_parser.c:80 at wm_parse_output(): ERROR: Test_message: ERROR
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_parser.c:76 at wm_parse_output(): INFO: Storage: Authenticating.
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_parser.c:80 at wm_parse_output(): ERROR: Error: The authentication file could not be opened: [Errno 2] No such file or directory: '/var/ossec/wodles_credentials/azure_storage_credentials'
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_azure.c:369 at wm_azure_storage(): INFO: Finished Storage log collection for container 'frameworktestcontainer'.
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_azure.c:94 at wm_azure_main(): INFO: Finished Storage log collection for 'azure-activity'.
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_azure.c:100 at wm_azure_main(): DEBUG: Fetching logs finished.
2023/12/20 18:19:20 wazuh-modulesd:azure-logs[3669] wm_azure.c:70 at wm_azure_main(): DEBUG: Sleeping until: 2023/12/20 18:29:19
Finally unit test were relaunched to ensure wodles its working,
(unit-test) eduardoleon@pop-os:~/git/wazuh(16739-refactor-azure-logs)$ pytest wodles/
=========================== test session starts ===========================
platform linux -- Python 3.9.16, pytest-7.4.2, pluggy-0.13.1
rootdir: /home/eduardoleon/git/wazuh
plugins: anyio-3.6.2, aiohttp-1.0.4, trio-0.7.0, html-3.0.0, asyncio-0.18.1, cov-3.0.0, metadata-2.0.2
asyncio: mode=auto
collected 852 items
wodles/aws/tests/test_aws_bucket.py ............................... [ 3%]
................................................................... [ 11%]
................................................................... [ 19%]
............................................ [ 24%]
wodles/aws/tests/test_aws_s3.py .................. [ 26%]
wodles/aws/tests/test_aws_service.py .... [ 27%]
wodles/aws/tests/test_cloudtrail.py .. [ 27%]
wodles/aws/tests/test_cloudwatchlogs.py ........................... [ 30%]
.......................... [ 33%]
wodles/aws/tests/test_config.py ................................... [ 37%]
........................................... [ 42%]
wodles/aws/tests/test_guardduty.py ................. [ 44%]
wodles/aws/tests/test_inspector.py ...... [ 45%]
wodles/aws/tests/test_load_balancers.py ............ [ 46%]
wodles/aws/tests/test_s3_log_handler.py .... [ 47%]
wodles/aws/tests/test_server_access.py ............................ [ 50%]
..... [ 51%]
wodles/aws/tests/test_sqs_queue.py ....... [ 51%]
wodles/aws/tests/test_tools.py ............................... [ 55%]
wodles/aws/tests/test_umbrella.py ...... [ 56%]
wodles/aws/tests/test_vpcflow.py ............................ [ 59%]
wodles/aws/tests/test_waf.py ....... [ 60%]
wodles/aws/tests/test_wazuh_integration.py ........................ [ 63%]
................................................................... [ 71%]
........... [ 72%]
wodles/azure/tests/test_azure.py .................................. [ 76%]
................................................................... [ 84%]
............ [ 85%]
wodles/azure/tests/test_orm.py ............................... [ 89%]
wodles/docker-listener/tests/test_docker_listener.py .............. [ 90%]
..... [ 91%]
wodles/gcloud/tests/test_bucket.py ................................ [ 95%]
. [ 95%]
wodles/gcloud/tests/test_gcloud.py ......... [ 96%]
wodles/gcloud/tests/test_integration.py ........ [ 97%]
wodles/gcloud/tests/test_subscriber.py .............. [ 99%]
wodles/gcloud/tests/test_tools.py ........ [100%]
============================ warnings summary =============================
../../.pyenv/versions/3.9.16/envs/unit-test/lib/python3.9/site-packages/pytest_aiohttp/plugin.py:28
/home/eduardoleon/.pyenv/versions/3.9.16/envs/unit-test/lib/python3.9/site-packages/pytest_aiohttp/plugin.py:28: DeprecationWarning: The 'asyncio_mode' is 'legacy', switching to 'auto' for the sake of pytest-aiohttp backward compatibility. Please explicitly use 'asyncio_mode=strict' or 'asyncio_mode=auto' in pytest configuration file.
config.issue_config_time_warning(LEGACY_MODE, stacklevel=2)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================== 852 passed, 1 warning in 3.08s ======================
The logger has been successfully integrated and the module is logging the expected messages.
Description
As described in #16717, To standardize the logging a refactor in this cloud module needs to be done to integrate the logging tool developed in the mentioned issue.
Logging refactor on the module should be done to log relevant information about the process.
Unit tests should be reviewed as well to adapt to the new logging method.
Tasks