Modify AWS module to process logs from WAF v2

GGP1 commented 5 months ago

Epic
#23361

Description

Our current integration with AWS WAF is only extracting records through Kinesis.

In this issue, we should perform the modifications necessary to our AWS module to fetch the records being generated on AWS WAF v2, following the path

<bucket_name>/<prefix>/AWSLogs/<suffix>/<account_id>/WAFLogs/<region>/<waf_name>/<year>/<month>/<day>/<hh>/<mm>

[!Note] The Migrating your AWS WAF Classic resources to AWS WAF guide might be useful.

Checks

The following elements have been updated or reviewed (should also be checked if no modification is required):

[x] Tests (unit tests, API integration tests).
[x] Changelog.
[x] Documentation.
[x] Integration test mapping (using api/test/integration/mapping/_test_mapping.py).

Selutario commented 5 months ago

https://github.com/wazuh/internal-devel-requests/issues/967#issuecomment-2003638945

javiersanchz commented 4 months ago

Update

I have been reading about the migration changes
I have been reviewing all the research that was done on it at https://github.com/wazuh/wazuh-documentation/issues/5808
I have been looking at changes in the waf.py code and reviewing about it. I started with the reproduction about it.

javiersanchz commented 4 months ago

Update

I'm still working on the issue, I've been reproducing the case mentioned about the discrepancy in routes when the record comes from WAF classic or WAF v2 when logging in directly to an S3 instead of through Kinesis.
I've been making changes to the code to address this

javiersanchz commented 4 months ago

Update

I have been requesting all the necessary permissions to create a WAF v2, to be able to create rules, add an S3 bucket...
I have been testing generating the logs manually with the following output in the /var/ossec/logs/alerts/alerts.json:

{"timestamp":"2024-04-29T11:38:31.092+0000","rule":{"level":3,"description":"AWS WAF - Blocked request.","id":"80442","firedtimes":3,"mail":false,"groups":["amazon","aws","aws_waf","aws_waf_block"]},"agent":{"id":"000","name":"wazuh-master"},"manager":{"name":"wazuh-master"},"id":"xxx.822910","cluster":{"name":"wazuh","node":"master-node"},"decoder":{"name":"json"},"data":{"integration":"aws","aws":{"log_info":{"log_file":"2019/10/23/11/aws-waf-logs-delivery-stream-1-2019-10-23-11-32-48-7xxd1f-bfed-4b00-9f5e-88ce44718194","s3bucket":"wazuh-aws-wodle-waf"},"timestamp":"1576280412771.000000","formatVersion":"1","webaclId":"arn:aws:wafv2:ap-southeast-2:1xxx5:regional/webacl/test/111","terminatingRuleId":"STMTest_SQLi_XSS","terminatingRuleType":"REGULAR","action":"BLOCK","terminatingRuleMatchDetails":{"conditionType":"SQL_INJECTION","location":"UNKNOWN","matchedData":["10","AND","1"]},"httpSourceName":"ALB","httpSourceId":"alb","ruleGroupList":[],"rateBasedRuleList":[],"nonTerminatingMatchingRules":[],"httpRequest":{"clientIp":"1.1.1.1","country":"AU","httpVersion":"HTTP/1.1","httpMethod":"POST","requestId":"null"},"labels":{"name":"value"},"source":"waf"}},"location":"Wazuh-AWS"}

I've been creating an AWS WAF v2, associating a bucket, and creating rules to generate alerts:

I did some research on how to generate v2 logs.

javiersanchz commented 4 months ago

Update

I continue working on the migration to v2, I'm unable to generate the logs on an AWS WAF v2 (it might also be due to lack of permissions).
The S3 bucket associated with it still isn't receiving any kind of log:

root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2  --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
DEBUG: +++ Marker: 2019/10/22
DEBUG: +++ No logs to process in bucket: aws-waf-logs-wodle-v2
DEBUG: +++ DB Maintenance

javiersanchz commented 4 months ago

Update

I have been setting up a real WAF environment for log generation in the S3 bucket:

Created a new EC2 instance, adding it to a target group.
Created and configured an ACL (Access Control List).
Created and configured an ALB (AWS Load Balancer).
Associated the web ACL with the ALB through AWS WAF.
I have been configuring rules in the WAF so that when activated, they generate records that can be seen in the created S3 bucket and view the generated path.

javiersanchz commented 4 months ago

Update

After setting up the environment, our AWS WAF v2 generated the path in our S3 bucket with the following route:

Log format:

{"timestamp":1714735190413,"formatVersion":1,"webaclId":"arn:aws:wafv2:us-east-1:xxxxxx:regional/webacl/AWS-WAF-V2/xxxxxxx817d-f47add445857","terminatingRuleId":"Dont-allow-HTTP-GET-and-POST","terminatingRuleType":"REGULAR","action":"BLOCK","terminatingRuleMatchDetails":[],"httpSourceName":"ALB","httpSourceId":"xxxx422-app/ABL-WAF-V2/xxxxxc1b5f","ruleGroupList":[],"rateBasedRuleList":[],"nonTerminatingMatchingRules":[],"requestHeadersInserted":null,"responseCodeSent":402,"httpRequest":{"clientIp":"46.174.191.28","country":"UA","headers":[{"name":"User-Agent","value":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; AS; rv:11.0) like Gecko"},{"name":"Accept","value":"*/*"}],"uri":"/","args":"","httpVersion":"HTTP/1.0","httpMethod":"GET","requestId":"xxxxxxxx0713ede17e"}}

The path generated by WAF v2 would be the following:

<bucket_name>/<prefix>/AWSLogs/<suffix>/<account_id>/WAFLogs/<region>/<waf_name>/<year>/<month>/<day>/<hh>/<mm>

The route should be adapted on the AWSWAFBucket class to obtain the generated logs

javiersanchz commented 4 months ago

Update

We talked with the rest of the team about which path should be processed within waf.py:
- Path generated by WAF for storing records directly in S3: <bucket_name>/<prefix>/AWSLogs/<suffix>/<account_id>/WAFLogs/<region>/<waf_name>/<year>/<month>/<day>/<hh>/<mm>
- Path generated by WAF using Kinesis: <bucket_name>/<prefix>/<year>/<month>/<day>
These paths are almost the same as those from GuardDuty:
- Path generated by GuardDuty for storing records directly in S3: <bucket_name>/<prefix>/AWSLogs/<suffix>/<account_id>/WAFLogs/<region>/<waf_name>/<year>/<month>/<day>
- Path generated by GuardDuty using Kinesis: <bucket_name>/<prefix>/<year>/<month>/<day>
As you can see, the path using Kinesis is the same, and the path generated directly in the S3 bucket only differs in that WAF adds /<hh>/<mm>
Therefore, the decision was made to follow more or less the same procedure, processing both types of paths for WAF by issuing a message similar to what GuardDuty has been doing regarding the deprecation of Kinesis:

https://github.com/wazuh/wazuh/blob/1e51c2d9ad140ba3dcdf60fdb9c5a79c43c8ffb3/wodles/aws/buckets_s3/guardduty.py#L12-L17

javiersanchz commented 4 months ago

Update

It seems that when launching the WAF bucket, it's not processing the logs properly because the accessed path is only up to the day:

<bucket_name>/<prefix>/AWSLogs/<suffix>/<account_id>/WAFLogs/<region>/<waf_name>/<year>/<month>/<day>

As can be seen here:

root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2  --aws_profile default --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
DEBUG: +++ Working on 567970947422 - us-east-1
DEBUG: +++ Marker: AWSLogs/567970947422/WAFLogs/us-east-1/2024/05/06
DEBUG: +++ Unexpected error: 'bucket'
ERROR: Unexpected error querying/working with objects in S3: 'bucket'

I've been testing changes in aws_bucket, specifically in the function responsible for it, get_creation_date, but it still uses the same path only up to the <day>.

javiersanchz commented 4 months ago

Update

After discussing with the rest of the team about the WAF operation, it has been decided to create an epic https://github.com/wazuh/wazuh/issues/23361 to split the workload, as it is more than anticipated since the logging route differs from the rest of AWS integrations.
This issue will focus on obtaining the logs generated in the S3 bucket by WAF, either natively or through Kinesis, as well as unit testing

javiersanchz commented 3 months ago

Update

The necessary changes were made to obtain records of WAF v2 route, and the following tests were carried out:

WAF native :

root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2  --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
DEBUG: +++ Working on 567970947422 - us-east-1
DEBUG: +++ Marker: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2019/10/22
DEBUG: ++ Found new log: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/10/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1110Z_a195076a.log.gz
DEBUG: ++ Found new log: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz
DEBUG: +++ DB Maintenance

As you can see, the logs belonging to WAF v2 routes were received. Next, the output of these logs has been verified at:

Alerts.log :

root@wazuh-master:/# grep '567970947422' /var/ossec/logs/alerts/alerts.log
{"integration": "aws", "aws": {"log_info": {"aws_account_alias": "", "log_file": "AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/10/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1110Z_a195076a.log.gz", "s3bucket": "aws-waf-logs-wodle-v2"}, "timestamp": 1714734687613, "formatVersion": 1, "webaclId": "arn:aws:wafv2:us-east-1:567970947422:regional/webacl/AWS-WAF-V2/98f8c0f9-6ec8-4c11-817d-xxxxxx", "terminatingRuleId": "Dont-allow-HTTP-GET-and-POST", "terminatingRuleType": "REGULAR", "action": "BLOCK", "terminatingRuleMatchDetails": [], "httpSourceName": "ALB", "httpSourceId": "567970947422-app/ABL-WAF-V2/xxxxx", "ruleGroupList": [], "rateBasedRuleList": [], "nonTerminatingMatchingRules": [], "responseCodeSent": 402, "httpRequest": {"clientIp": "87.236.176.171", "country": "GB", "headers": {"Host": "35.153.251.153", "User-Agent": "Mozilla/5.0 (compatible; InternetMeasurement/1.0; +https://internet-measurement.com/)", "Connection": "close", "Accept": "*/*", "Accept-Encoding": "gzip"}, "uri": "/", "args": "", "httpVersion": "HTTP/1.1", "httpMethod": "GET", "requestId": "1-6634c65f-xxxxxxxxx"}, "source": "waf"}}
aws.log_info.log_file: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/10/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1110Z_a195076a.log.gz
{"integration": "aws", "aws": {"log_info": {"aws_account_alias": "", "log_file": "AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz", "s3bucket": "aws-waf-logs-wodle-v2"}, "timestamp": 1714735190413, "formatVersion": 1, "webaclId": "arn:aws:wafv2:us-east-1:567970947422:regional/webacl/AWS-WAF-V2/98f8c0f9-6ec8-4c11-817d-f47add445857", "terminatingRuleId": "Dont-allow-HTTP-GET-and-POST", "terminatingRuleType": "REGULAR", "action": "BLOCK", "terminatingRuleMatchDetails": [], "httpSourceName": "ALB", "httpSourceId": "567970947422-app/ABL-WAF-V2/xxxxxxx", "ruleGroupList": [], "rateBasedRuleList": [], "nonTerminatingMatchingRules": [], "responseCodeSent": 402, "httpRequest": {"clientIp": "46.174.191.28", "country": "UA", "headers": {"User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; AS; rv:11.0) like Gecko", "Accept": "*/*"}, "uri": "/", "args": "", "httpVersion": "HTTP/1.0", "httpMethod": "GET", "requestId": "xxxxxxxxxxxxxxxx"}, "source": "waf"}}
aws.log_info.log_file: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz

Alerts.json :

{"timestamp":"2024-05-13T11:22:26.600+0000","rule":{"level":3,"description":"AWS WAF - Blocked request.","id":"xxx","firedtimes":1,"mail":false,"groups":["amazon","aws","aws_waf","aws_waf_block"]},"agent":{"id":"000","name":"wazuh-master"},"manager":{"name":"wazuh-master"},"id":"xxxxxx","cluster":{"name":"wazuh","node":"master-node"},"decoder":{"name":"json"},"data":{"integration":"aws","aws":{"log_info":{"log_file":"AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/10/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1110Z_a195076a.log.gz","s3bucket":"aws-waf-logs-wodle-v2"},"timestamp":"1714734687613.000000","formatVersion":"1","webaclId":"arn:aws:wafv2:us-east-1:567970947422:regional/webacl/AWS-WAF-V2/98f8c0f9-6ec8-4c11-817d-f47add445857","terminatingRuleId":"Dont-allow-HTTP-GET-and-POST","terminatingRuleType":"REGULAR","action":"BLOCK","terminatingRuleMatchDetails":[],"httpSourceName":"ALB","httpSourceId":"567970947422-app/ABL-WAF-V2/27426a40f4ac1b5f","ruleGroupList":[],"rateBasedRuleList":[],"nonTerminatingMatchingRules":[],"responseCodeSent":"402","httpRequest":{"clientIp":"87.236.176.171","country":"GB","headers":{"Host":"35.153.251.153","User-Agent":"Mozilla/5.0 (compatible; InternetMeasurement/1.0; +https://internet-measurement.com/)","Connection":"close","Accept":"*/*","Accept-Encoding":"gzip"},"uri":"/","httpVersion":"HTTP/1.1","httpMethod":"GET","requestId":"1xxxxxxxxx"},"source":"waf"}},"location":"Wazuh-AWS"}
{"timestamp":"2024-05-13T11:22:26.885+0000","rule":{"level":3,"description":"AWS WAF - Blocked request.","id":"xxxx","firedtimes":2,"mail":false,"groups":["amazon","aws","aws_waf","aws_waf_block"]},"agent":{"id":"000","name":"wazuh-master"},"manager":{"name":"wazuh-master"},"id":"xxxxxx","cluster":{"name":"wazuh","node":"master-node"},"decoder":{"name":"json"},"data":{"integration":"aws","aws":{"log_info":{"log_file":"AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz","s3bucket":"aws-waf-logs-wodle-v2"},"timestamp":"1714735190413.000000","formatVersion":"1","webaclId":"arn:aws:wafv2:us-east-1:567970947422:regional/webacl/AWS-WAF-V2/98f8c0f9-6ec8-4c11-817d-f47add445857","terminatingRuleId":"Dont-allow-HTTP-GET-and-POST","terminatingRuleType":"REGULAR","action":"BLOCK","terminatingRuleMatchDetails":[],"httpSourceName":"ALB","httpSourceId":"567970947422-app/ABL-WAF-V2/27426a40f4ac1b5f","ruleGroupList":[],"rateBasedRuleList":[],"nonTerminatingMatchingRules":[],"responseCodeSent":"402","httpRequest":{"clientIp":"46.174.191.28","country":"UA","headers":{"User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; AS; rv:11.0) like Gecko","Accept":"*/*"},"uri":"/","httpVersion":"HTTP/1.0","httpMethod":"GET","requestId":"xxxxxxxx"},"source":"waf"}},"location":"Wazuh-AWS"}

WAF kinesis :

The operation of WAF Kinesis was also verified, including the deprecation message starting from version 5.0:

root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket wazuh-aws-wodle-waf  --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
The functionality to process WAF logs stored in S3 via Kinesis was deprecated in 5.0. Consider configuring WAF to store its logs directly in an S3 bucket instead. Check https://documentation.wazuh.com/current/amazon/services/supported-services/waf.html for more information.
DEBUG: +++ Marker: 2019/10/22
DEBUG: ++ Found new log: 2019/10/23/10/aws-waf-logs-delivery-stream-1-2019-10-23-10-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194
DEBUG: ++ Found new log: 2019/10/23/11/aws-waf-logs-delivery-stream-1-2019-10-23-11-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194
DEBUG: ++ Found new log: 2022/06/03/aws-waf-logs-delivery-stream-1-2022-06-03-10-32-48-792c6d1f-bfed-4b00-9f5e-88ce4471fake
DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-1
DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-2
DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-3
DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194
DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-No-Modifications
DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-multiple-values-in-ruleGroupList
DEBUG: ++ Found new log: 2023/12/08/17/aws-waf-logs-delivery-stream-1-2023-12-08-17-32-48-No-Modifications
DEBUG: ++ Found new log: 2023/12/08/17/aws-waf-logs-delivery-stream-1-2023-12-08-17-32-48-multiple-values-in-ruleGroupList
DEBUG: +++ DB Maintenance

The new unit tests for AWS WAF are pending creation.

javiersanchz commented 3 months ago

Update

The tests related to WAF were updated and added:

(unittest-env) wazuh@javier:~/Git/wazuh$ PYTHONPATH=/home/wazuh/Git/wazuh/api:/home/wazuh/Git/wazuh/framework python3 -m pytest wodles/aws/tests/test_waf.py -v
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.10.12, pytest-7.3.1, pluggy-1.4.0 -- /home/wazuh/venv/unittest-env/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.10.12', 'Platform': 'Linux-6.5.0-17-generic-x86_64-with-glibc2.35', 'Packages': {'pytest': '7.3.1', 'pluggy': '1.4.0'}, 'Plugins': {'anyio': '4.3.0', 'aiohttp': '1.0.4', 'trio': '0.8.0', 'html': '2.1.1', 'metadata': '3.1.0', 'asyncio': '0.18.1', 'tavern': '1.23.5'}}
rootdir: /home/wazuh/Git/wazuh/wodles/aws/tests
configfile: pytest.ini
plugins: anyio-4.3.0, aiohttp-1.0.4, trio-0.8.0, html-2.1.1, metadata-3.1.0, asyncio-0.18.1, tavern-1.23.5
asyncio: mode=auto
collected 15 items                                                                                                                                                                                                

wodles/aws/tests/test_waf.py::test_aws_waf_bucket_initializes_properly PASSED                                                                                                                               [  6%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-False] PASSED                                                [ 13%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-True] PASSED                                                 [ 20%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-invalid-json-True] PASSED                                    [ 26%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-wrong-structure-True] PASSED                                 [ 33%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file_handles_exception_on_invalid_argument[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-invalid-json-False-SystemExit] PASSED [ 40%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_load_information_from_file_handles_exception_on_invalid_argument[/home/wazuh/Git/wazuh/wodles/aws/tests/data/log_files/WAF/aws-waf-wrong-structure-False-SystemExit] PASSED [ 46%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_check_waf_type[object_list0-True] PASSED                                                                                                                  [ 53%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_check_waf_type[object_list1-False] PASSED                                                                                                                 [ 60%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_check_waf_type_handles_exceptions PASSED                                                                                                                  [ 66%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_get_service_prefix PASSED                                                                                                                                 [ 73%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_get_base_prefix[True] PASSED                                                                                                                              [ 80%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_get_base_prefix[False] PASSED                                                                                                                             [ 86%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_iter_regions_and_accounts[True] PASSED                                                                                                                    [ 93%]
wodles/aws/tests/test_waf.py::test_aws_waf_bucket_iter_regions_and_accounts[False] PASSED                                                                                                                   [100%]

=============================================================================================== 15 passed in 0.27s ================================================================================================

(unittest-env) wazuh@javier:~/Git/wazuh$ PYTHONPATH=/home/wazuh/Git/wazuh/api:/home/wazuh/Git/wazuh/framework python3 -m pytest wodles/aws/tests
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.10.12, pytest-7.3.1, pluggy-1.4.0
rootdir: /home/wazuh/Git/wazuh/wodles/aws/tests
configfile: pytest.ini
plugins: anyio-4.3.0, aiohttp-1.0.4, trio-0.8.0, html-2.1.1, metadata-3.1.0, asyncio-0.18.1, tavern-1.23.5
asyncio: mode=auto
collected 598 items                                                                                                                                                                                               

wodles/aws/tests/test_aws_bucket.py ....................................................................................................................................................................... [ 27%]
....................................                                                                                                                                                                        [ 33%]
wodles/aws/tests/test_aws_s3.py ....................                                                                                                                                                        [ 37%]
wodles/aws/tests/test_aws_service.py ....                                                                                                                                                                   [ 37%]
wodles/aws/tests/test_cloudtrail.py ..                                                                                                                                                                      [ 38%]
wodles/aws/tests/test_cloudwatchlogs.py .....................................................                                                                                                               [ 47%]
wodles/aws/tests/test_config.py ..............................................................................                                                                                              [ 60%]
wodles/aws/tests/test_guardduty.py .................                                                                                                                                                        [ 63%]
wodles/aws/tests/test_inspector.py ......                                                                                                                                                                   [ 64%]
wodles/aws/tests/test_load_balancers.py ............                                                                                                                                                        [ 66%]
wodles/aws/tests/test_s3_log_handler.py ................                                                                                                                                                    [ 68%]
wodles/aws/tests/test_server_access.py .................................                                                                                                                                    [ 74%]
wodles/aws/tests/test_sqs_message_processor.py ........                                                                                                                                                     [ 75%]
wodles/aws/tests/test_sqs_queue.py .......                                                                                                                                                                  [ 76%]
wodles/aws/tests/test_tools.py ..................................                                                                                                                                           [ 82%]
wodles/aws/tests/test_umbrella.py ......                                                                                                                                                                    [ 83%]
wodles/aws/tests/test_vpcflow.py .....................                                                                                                                                                      [ 86%]
wodles/aws/tests/test_waf.py ...............                                                                                                                                                                [ 89%]
wodles/aws/tests/test_wazuh_integration.py ...............................................................                                                                                                  [100%]

=============================================================================================== 598 passed in 2.94s ===============================================================================================

javiersanchz commented 3 months ago

Update

The requested changes were checked and the tests for WAF v2 were run again. As the changes were, errors were being generated when running the bucket with the following output:

root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2  --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10
DEBUG: Created Config object using profile: 'default' configuration
DEBUG: +++ Marker: AWSLogs/567970947422/WAFLogs/None/AWS-WAF-V2/2019/10/22
DEBUG: +++ No logs to process in bucket: aws-waf-logs-wodle-v2
DEBUG: +++ DB Maintenanc

As can be seen in the bucket, it is marking the region being iterated over as None. This issue arose due to the change introduced in the review https://github.com/wazuh/wazuh/pull/23397#discussion_r1606848711, without realizing that depending on the type of waf_type, it uses the method from one class or another.

This was corrected back to how it was before, the other requested changes were added, and the corresponding tests were updated:

Tests performed:

Tests performed

I run the WAF v2 bucket: ```console root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2 --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2 DEBUG: +++ Debug mode on - Level: 2 DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10 DEBUG: Created Config object using profile: 'default' configuration DEBUG: +++ Table does not exist; create DEBUG: +++ Working on 567970947422 - us-east-1 DEBUG: +++ Marker: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2019/10/22 DEBUG: ++ Found new log: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/10/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1110Z_a195076a.log.gz DEBUG: ++ Found new log: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz DEBUG: +++ DB Maintenance ``` I run the bucket again after processing the logs: ```console root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket aws-waf-logs-wodle-v2 --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2 DEBUG: +++ Debug mode on - Level: 2 DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10 DEBUG: Created Config object using profile: 'default' configuration DEBUG: +++ Working on 567970947422 - us-east-1 DEBUG: +++ Marker: AWSLogs/567970947422/WAFLogs/us-east-1/AWS-WAF-V2/2024/05/03/11/15/567970947422_waflogs_us-east-1_AWS-WAF-V2_20240503T1115Z_7409afbb.log.gz DEBUG: +++ No logs to process in bucket: 567970947422/us-east-1 DEBUG: +++ DB Maintenance ``` I run the Kinesis bucket: ```console root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 --bucket wazuh-aws-wodle-waf --aws_profile default --only_logs_after 2019-OCT-22 --regions us-east-1 --type waf --skip_on_error -d 2 DEBUG: +++ Debug mode on - Level: 2 DEBUG: No retries configuration found in profile config. Generating default configuration for retries: mode: standard - max_attempts: 10 DEBUG: Created Config object using profile: 'default' configuration The functionality to process WAF logs stored in S3 via Kinesis was deprecated in 5.0. Consider configuring WAF to store its logs directly in an S3 bucket instead. Check https://documentation.wazuh.com/current/amazon/services/supported-services/waf.html for more information. DEBUG: +++ Marker: 2019/10/22 DEBUG: ++ Found new log: 2019/10/23/10/aws-waf-logs-delivery-stream-1-2019-10-23-10-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194 DEBUG: ++ Found new log: 2019/10/23/11/aws-waf-logs-delivery-stream-1-2019-10-23-11-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194 DEBUG: ++ Found new log: 2022/06/03/aws-waf-logs-delivery-stream-1-2022-06-03-10-32-48-792c6d1f-bfed-4b00-9f5e-88ce4471fake ERROR: the 2022/06/03/aws-waf-logs-delivery-stream-1-2022-06-03-10-32-48-792c6d1f-bfed-4b00-9f5e-88ce4471fake file doesn't have the expected structure. DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-1 DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-2 DEBUG: ++ Found new log: 2022/11/30/11/aws-waf-logs-delivery-stream-1-2022-11-30-11-32-48-sample-3 DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-792c6d1f-bfed-4b00-9f5e-88ce44718194 DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-No-Modifications DEBUG: ++ Found new log: 2023/12/05/17/aws-waf-logs-delivery-stream-1-2023-12-05-17-32-48-multiple-values-in-ruleGroupList DEBUG: ++ Found new log: 2023/12/08/17/aws-waf-logs-delivery-stream-1-2023-12-08-17-32-48-No-Modifications DEBUG: ++ Found new log: 2023/12/08/17/aws-waf-logs-delivery-stream-1-2023-12-08-17-32-48-multiple-values-in-ruleGroupList DEBUG: +++ DB Maintenance ```

The alerts generated from the logs were also verified.

The related tests were checked:

Tests

```console (unittest-env) wazuh@javier:~/Git/wazuh$ PYTHONPATH=/home/wazuh/Git/wazuh/api:/home/wazuh/Git/wazuh/framework python3 -m pytest wodles/aws/tests/test_waf.py =============================================================================================== test session starts =============================================================================================== platform linux -- Python 3.10.12, pytest-7.3.1, pluggy-1.4.0 rootdir: /home/wazuh/Git/wazuh/wodles/aws/tests configfile: pytest.ini plugins: anyio-4.3.0, aiohttp-1.0.4, trio-0.8.0, html-2.1.1, metadata-3.1.0, asyncio-0.18.1, tavern-1.23.5 asyncio: mode=auto collected 15 items wodles/aws/tests/test_waf.py ............... [100%] =============================================================================================== 15 passed in 0.26s ================================================================================================ (unittest-env) wazuh@javier:~/Git/wazuh$ PYTHONPATH=/home/wazuh/Git/wazuh/api:/home/wazuh/Git/wazuh/framework python3 -m pytest wodles/aws/tests =============================================================================================== test session starts =============================================================================================== platform linux -- Python 3.10.12, pytest-7.3.1, pluggy-1.4.0 rootdir: /home/wazuh/Git/wazuh/wodles/aws/tests configfile: pytest.ini plugins: anyio-4.3.0, aiohttp-1.0.4, trio-0.8.0, html-2.1.1, metadata-3.1.0, asyncio-0.18.1, tavern-1.23.5 asyncio: mode=auto collected 598 items wodles/aws/tests/test_aws_bucket.py ....................................................................................................................................................................... [ 27%] .................................... [ 33%] wodles/aws/tests/test_aws_s3.py .................... [ 37%] wodles/aws/tests/test_aws_service.py .... [ 37%] wodles/aws/tests/test_cloudtrail.py .. [ 38%] wodles/aws/tests/test_cloudwatchlogs.py ..................................................... [ 47%] wodles/aws/tests/test_config.py .............................................................................. [ 60%] wodles/aws/tests/test_guardduty.py ................. [ 63%] wodles/aws/tests/test_inspector.py ...... [ 64%] wodles/aws/tests/test_load_balancers.py ............ [ 66%] wodles/aws/tests/test_s3_log_handler.py ................ [ 68%] wodles/aws/tests/test_server_access.py ................................. [ 74%] wodles/aws/tests/test_sqs_message_processor.py ........ [ 75%] wodles/aws/tests/test_sqs_queue.py ....... [ 76%] wodles/aws/tests/test_tools.py .................................. [ 82%] wodles/aws/tests/test_umbrella.py ...... [ 83%] wodles/aws/tests/test_vpcflow.py ..................... [ 86%] wodles/aws/tests/test_waf.py ............... [ 89%] wodles/aws/tests/test_wazuh_integration.py ............................................................... [100%] =============================================================================================== 598 passed in 2.59s =============================================================================================== ```

javiersanchz commented 3 months ago

Update

About the last requested changes:

Fixed syntax errors in the code.
Modified the get_full_prefix method to access the last key of the Contents directly and get the acl_name
The functionality of AWS WAF was tested again.
The tests were rerun, and I verified that they all passed.

wazuh / wazuh