search

prowler-cloud / prowler

Prowler is an Open Source Security tool for AWS, Azure, GCP and Kubernetes to do security assessments, audits, incident response, compliance, continuous monitoring, hardening and forensics readiness. Includes CIS, NIST 800, NIST CSF, CISA, FedRAMP, PCI-DSS, GDPR, HIPAA, FFIEC, SOC2, GXP, Well-Architected Security, ENS and more
https://prowler.com
Apache License 2.0
10.85k stars 1.54k forks source link

[Bug]: Prowler gets stuck/fails at step 49/281 #2458

Closed rweirtbg closed 1 year ago

rweirtbg commented 1 year ago

Steps to Reproduce

When I run prowler aws I get a good chunk of the way through and get stuck at step 49/281:

Expected behavior

No error

Actual Result with Screenshots or Logs

Screenshot 2023-06-06 at 2 39 43 PM

How did you install Prowler?

From pip package (pip install prowler)

Environment Resource

VM

OS used

Ubuntu 22.04 LTS

Prowler version

v3.5.3

Pip version

pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)

Context

No response

MrCloudSec commented 1 year ago

Hi @rweirtbg, thanks for contacting us! Could you execute it again by adding --log-level ERROR and send us the logs?

rweirtbg commented 1 year ago

These are the cloudwatch logs...nothing really stood out since we don't have services in these regions.

2023-06-06 21:51:27,027 [File: cloudwatch_service.py:67] [Module: cloudwatch_service] ERROR: eu-south-2 -- ClientError[46]: An error occurred (InvalidClientTokenId) when calling the DescribeAlarms operation: The security token included in the request is invalid. 2023-06-06 21:51:27,368 [File: cloudwatch_service.py:67] [Module: cloudwatch_service] ERROR: eu-south-1 -- ClientError[46]: An error occurred (InvalidClientTokenId) when calling the DescribeAlarms operation: The security token included in the request is invalid. 2023-06-06 21:51:27,434 [File: cloudwatch_service.py:67] [Module: cloudwatch_service] ERROR: me-central-1 -- ClientError[46]: An error occurred (InvalidClientTokenId) when calling the DescribeAlarms operation: The security token included in the request is invalid. 2023-06-06 21:51:27,466 [File: cloudwatch_service.py:67] [Module: cloudwatch_service] ERROR: eu-central-2 -- ClientError[46]: An error occurred (InvalidClientTokenId) when calling the DescribeAlarms operation: The security token included in the request is invalid. 2023-06-06 21:51:27,489 [File: cloudwatch_service.py:67] [Module: cloudwatch_service] ERROR: af-south-1 -- ClientError[46]: An error occurred (InvalidClientTokenId) when calling the DescribeAlarms operation: The security token included in the request is invalid. 2023-06-06 21:51:27,519 [File: cloudwatch_service.py:67] [Module: cloudwatch_service] ERROR: me-south-1 -- ClientError[46]: An error occurred (InvalidClientTokenId) when calling the DescribeAlarms operation: The security token included in the request is invalid. 2023-06-06 21:51:27,739 [File: cloudwatch_service.py:67] [Module: cloudwatch_service] ERROR: ap-southeast-4 -- ClientError[46]: An error occurred (InvalidClientTokenId) when calling the DescribeAlarms operation: The security token included in the request is invalid. 2023-06-06 21:51:27,745 [File: cloudwatch_service.py:67] [Module: cloudwatch_service] ERROR: ap-southeast-3 -- ClientError[46]: An error occurred (InvalidClientTokenId) when calling the DescribeAlarms operation: The security token included in the request is invalid. 2023-06-06 21:51:27,860 [File: cloudwatch_service.py:67] [Module: cloudwatch_service] ERROR: ap-east-1 -- ClientError[46]: An error occurred (InvalidClientTokenId) when calling the DescribeAlarms operation: The security token included in the request is invalid. 2023-06-06 21:51:27,895 [File: cloudwatch_service.py:67] [Module: cloudwatch_service] ERROR: ap-south-2 -- ClientError[46]: An error occurred (InvalidClientTokenId) when calling the DescribeAlarms operation: The security token included in the request is invalid. 2023-06-06 21:51:39,913 [File: cloudwatch_service.py:141] [Module: cloudwatch_service] ERROR: eu-south-1 -- UnrecognizedClientException[126]: An error occurred (UnrecognizedClientException) when calling the DescribeMetricFilters operation: The security token included in the request is invalid. 2023-06-06 21:51:40,018 [File: cloudwatch_service.py:141] [Module: cloudwatch_service] ERROR: eu-central-2 -- UnrecognizedClientException[126]: An error occurred (UnrecognizedClientException) when calling the DescribeMetricFilters operation: The security token included in the request is invalid. 2023-06-06 21:51:40,064 [File: cloudwatch_service.py:141] [Module: cloudwatch_service] ERROR: ap-east-1 -- UnrecognizedClientException[126]: An error occurred (UnrecognizedClientException) when calling the DescribeMetricFilters operation: The security token included in the request is invalid. 2023-06-06 21:51:40,084 [File: cloudwatch_service.py:141] [Module: cloudwatch_service] ERROR: ap-southeast-4 -- UnrecognizedClientException[126]: An error occurred (UnrecognizedClientException) when calling the DescribeMetricFilters operation: The security token included in the request is invalid. 2023-06-06 21:51:40,112 [File: cloudwatch_service.py:141] [Module: cloudwatch_service] ERROR: eu-south-2 -- UnrecognizedClientException[126]: An error occurred (UnrecognizedClientException) when calling the DescribeMetricFilters operation: The security token included in the request is invalid. 2023-06-06 21:51:40,217 [File: cloudwatch_service.py:141] [Module: cloudwatch_service] ERROR: af-south-1 -- UnrecognizedClientException[126]: An error occurred (UnrecognizedClientException) when calling the DescribeMetricFilters operation: The security token included in the request is invalid. 2023-06-06 21:51:40,255 [File: cloudwatch_service.py:141] [Module: cloudwatch_service] ERROR: me-central-1 -- UnrecognizedClientException[126]: An error occurred (UnrecognizedClientException) when calling the DescribeMetricFilters operation: The security token included in the request is invalid. 2023-06-06 21:51:40,336 [File: cloudwatch_service.py:141] [Module: cloudwatch_service] ERROR: me-south-1 -- UnrecognizedClientException[126]: An error occurred (UnrecognizedClientException) when calling the DescribeMetricFilters operation: The security token included in the request is invalid. 2023-06-06 21:51:40,424 [File: cloudwatch_service.py:141] [Module: cloudwatch_service] ERROR: ap-south-2 -- UnrecognizedClientException[126]: An error occurred (UnrecognizedClientException) when calling the DescribeMetricFilters operation: The security token included in the request is invalid. 2023-06-06 21:51:40,638 [File: cloudwatch_service.py:141] [Module: cloudwatch_service] ERROR: ap-southeast-3 -- UnrecognizedClientException[126]: An error occurred (UnrecognizedClientException) when calling the DescribeMetricFilters operation: The security token included in the request is invalid. 2023-06-06 21:51:42,387 [File: cloudwatch_service.py:173] [Module: cloudwatch_service] ERROR: eu-central-2 -- UnrecognizedClientException[151]: An error occurred (UnrecognizedClientException) when calling the DescribeLogGroups operation: The security token included in the request is invalid. 2023-06-06 21:51:42,498 [File: cloudwatch_service.py:173] [Module: cloudwatch_service] ERROR: eu-south-1 -- UnrecognizedClientException[151]: An error occurred (UnrecognizedClientException) when calling the DescribeLogGroups operation: The security token included in the request is invalid.

rweirtbg commented 1 year ago

I re-ran prowler using the following to isolate the issue:

prowler aws -f us-east-1 --log-level ERROR --services cloudwatch

It just hangs at 0% showing no errors.

MrCloudSec commented 1 year ago

Thanks for the logs @rweirtbg. Could you repeat the previous command using INFO level? prowler aws -f us-east-1 --log-level INFO --services cloudwatch And also the following, please: prowler aws -f us-east-1 --log-level INFO --services cloudwatch -e cloudwatch_log_group_no_secrets_in_logs

rweirtbg commented 1 year ago

In both cases it seems to get stuck at:

INFO: CloudWatch Logs - Describing log groups...

prowler aws -f us-east-1 --log-level INFO --services cloudwatch

2023-06-07 18:05:40,844 [File: check.py:438]    [Module: check]  WARNING: Your session file descriptors limit (1024 open files) is below 4096. We recommend to increase it to avoid errors. Solve it running this command `ulimit -n 4096`. For more info visit https://docs.prowler.cloud/en/latest/troubleshooting/
Executing 19 checks, please wait...

2023-06-07 18:05:41,107 [File: cloudtrail_service.py:49]        [Module: cloudtrail_service]     INFO: Cloudtrail - Getting trails...
2023-06-07 18:05:41,470 [File: cloudtrail_service.py:96]        [Module: cloudtrail_service]     INFO: Cloudtrail - Getting trail status
2023-06-07 18:05:41,513 [File: cloudtrail_service.py:147]       [Module: cloudtrail_service]     INFO: Cloudtrail - Getting trail insihgt selectors...
2023-06-07 18:05:41,552 [File: cloudtrail_service.py:114]       [Module: cloudtrail_service]     INFO: Cloudtrail - Getting event selector
2023-06-07 18:05:41,903 [File: cloudtrail_service.py:188]       [Module: cloudtrail_service]     INFO: CloudTrail - List Tags...
2023-06-07 18:05:42,029 [File: cloudwatch_service.py:43]        [Module: cloudwatch_service]     INFO: CloudWatch - Describing alarms...
2023-06-07 18:05:42,714 [File: cloudwatch_service.py:72]        [Module: cloudwatch_service]     INFO: CloudWatch - List Tags...
2023-06-07 18:05:47,685 [File: cloudwatch_service.py:121]       [Module: cloudwatch_service]     INFO: CloudWatch Logs - Describing metric filters...
2023-06-07 18:05:48,037 [File: cloudwatch_service.py:146]       [Module: cloudwatch_service]     INFO: CloudWatch Logs - Describing log groups...
-> Scanning cloudwatch service |⚠︎                                       | (!) 0/19 [0%] in 5:23.9

prowler aws -f us-east-1 --log-level INFO --services cloudwatch -e cloudwatch_log_group_no_secrets_in_logs

2023-06-07 18:11:15,961 [File: check.py:438]    [Module: check]  WARNING: Your session file descriptors limit (1024 open files) is below 4096. We recommend to increase it to avoid errors. Solve it running this command `ulimit -n 4096`. For more info visit https://docs.prowler.cloud/en/latest/troubleshooting/
Executing 18 checks, please wait...

2023-06-07 18:11:16,039 [File: cloudtrail_service.py:49]        [Module: cloudtrail_service]     INFO: Cloudtrail - Getting trails...
2023-06-07 18:11:16,223 [File: cloudtrail_service.py:96]        [Module: cloudtrail_service]     INFO: Cloudtrail - Getting trail status
2023-06-07 18:11:16,256 [File: cloudtrail_service.py:147]       [Module: cloudtrail_service]     INFO: Cloudtrail - Getting trail insihgt selectors...
2023-06-07 18:11:16,281 [File: cloudtrail_service.py:114]       [Module: cloudtrail_service]     INFO: Cloudtrail - Getting event selector
2023-06-07 18:11:16,487 [File: cloudtrail_service.py:188]       [Module: cloudtrail_service]     INFO: CloudTrail - List Tags...
2023-06-07 18:11:16,594 [File: cloudwatch_service.py:43]        [Module: cloudwatch_service]     INFO: CloudWatch - Describing alarms...
2023-06-07 18:11:17,269 [File: cloudwatch_service.py:72]        [Module: cloudwatch_service]     INFO: CloudWatch - List Tags...
2023-06-07 18:11:22,111 [File: cloudwatch_service.py:121]       [Module: cloudwatch_service]     INFO: CloudWatch Logs - Describing metric filters...
2023-06-07 18:11:22,454 [File: cloudwatch_service.py:146]       [Module: cloudwatch_service]     INFO: CloudWatch Logs - Describing log groups...
-> Scanning cloudwatch service |                                        | | 0/18 [0%] in 1:01
MrCloudSec commented 1 year ago

Thanks for the logs @rweirtbg. How many CloudWatch log groups do you have? Can you try it again after using ulimit -n 4096 command, please?

rweirtbg commented 1 year ago

Hi @sergargar ,

That seems to have fixed it!

It looks like we have 285070 log groups. 😳 It looks like that scan will take about 5 days to complete...is there any way to speed this up?

2023-06-08 01:21:33,352 [File: cloudtrail_service.py:49] [Module: cloudtrail_service] INFO: Cloudtrail - Getting trails... 2023-06-08 01:21:33,557 [File: cloudtrail_service.py:96] [Module: cloudtrail_service] INFO: Cloudtrail - Getting trail status 2023-06-08 01:21:33,603 [File: cloudtrail_service.py:147] [Module: cloudtrail_service] INFO: Cloudtrail - Getting trail insihgt selectors... 2023-06-08 01:21:33,635 [File: cloudtrail_service.py:114] [Module: cloudtrail_service] INFO: Cloudtrail - Getting event selector 2023-06-08 01:21:33,835 [File: cloudtrail_service.py:188] [Module: cloudtrail_service] INFO: CloudTrail - List Tags... 2023-06-08 01:21:33,898 [File: cloudwatch_service.py:43] [Module: cloudwatch_service] INFO: CloudWatch - Describing alarms... 2023-06-08 01:21:34,366 [File: cloudwatch_service.py:72] [Module: cloudwatch_service] INFO: CloudWatch - List Tags... 2023-06-08 01:21:39,020 [File: cloudwatch_service.py:121] [Module: cloudwatch_service] INFO: CloudWatch Logs - Describing metric filters... 2023-06-08 01:21:39,197 [File: cloudwatch_service.py:146] [Module: cloudwatch_service] INFO: CloudWatch Logs - Describing log groups... 2023-06-08 01:49:22,655 [File: cloudwatch_service.py:184] [Module: cloudwatch_service] INFO: CloudWatch Logs - Retrieving log events for 285070 log groups in us-east-1... 2023-06-08 01:49:47,009 [File: cloudwatch_service.py:198] [Module: cloudwatch_service] INFO: CloudWatch Logs - Retrieved log events for 10/285070 log groups in us-east-1... 2023-06-08 01:50:07,785 [File: cloudwatch_service.py:198] [Module: cloudwatch_service] INFO: CloudWatch Logs - Retrieved log events for 20/285070 log groups in us-east-1... 2023-06-08 01:50:30,643 [File: cloudwatch_service.py:198] [Module: cloudwatch_service] INFO: CloudWatch Logs - Retrieved log events for 30/285070 log groups in us-east-1... 2023-06-08 01:50:52,615 [File: cloudwatch_service.py:198] [Module: cloudwatch_service] INFO: CloudWatch Logs - Retrieved log events for 40/285070 log groups in us-east-1...

MrCloudSec commented 1 year ago

Great! You can speed it up by not getting all the log events for the check cloudwatch_log_group_no_secrets_in_logs using the following flag: -e cloudwatch_log_group_no_secrets_in_logs

MrCloudSec commented 1 year ago

@rweirtbg feel free to reopen the issue if you encounter any other problem. With the command above it should speed it up 💪🏼 Thanks for using Prowler!

rweirtbg commented 1 year ago

@sergargar So that exclusion (-e cloudwatch_log_group_no_secrets_in_logs) did not solve the issue. We were able to get the scan to run through completely, but we ended up with 285413 Cloudwatch findings and a 1.06GB HTML report.

MrCloudSec commented 1 year ago

@rweirtbg try to exclude the Cloudwatch service with --excluded-services cloudwatch to eliminate the 285413 Cloudwatch findings.