Fetch errors since upgrade to 5.13.0

[GijsbertVanOeveren]

Recently we upgraded to scoutsuite 5.13.0 now running on python3.9

Since then we experience various fetch errors we have never encountered before.

Command used: scout aws --report-name ${ACCOUNTALIAS} --debug --report-dir /var/www/html/ -f --logfile log/${ACCOUNTALIAS}.log --quiet --no-browser -r eu-west-1 eu-central us-east-1 --max-workers 3 --debug

Log output:

2023-07-24 12:02:33 scout[3328] ERROR rds.py L168: Failed fetching DB parameters for tf-pg-mariadb-10-6-wikidb: An error occurred (Throttling) when calling the DescribeDBParameters operation (reached max retries: 4): Rate exceeded Traceback (most recent call last): File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/facade/rds.py", line 168, in _get_and_set_db_parameters parameters = await AWSFacadeUtils.get_all_pages( File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/facade/utils.py", line 27, in get_all_pages results = await AWSFacadeUtils.get_multiple_entities_from_all_pages( File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/facade/utils.py", line 57, in get_multiple_entities_from_all_pages return await run_concurrently(lambda: AWSFacadeUtils._get_all_pages_from_paginator(paginator, entities)) File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/utils.py", line 27, in run_concurrently return await run_function_concurrently(function) File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, self.kwargs) File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/facade/utils.py", line 57, in return await run_concurrently(lambda: AWSFacadeUtils._get_all_pages_from_paginator(paginator, entities)) File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/facade/utils.py", line 73, in _get_all_pages_from_paginator for page in paginator: File "/opt/scoutsuite/lib/python3.9/site-packages/botocore/paginate.py", line 269, in iter response = self._make_request(current_kwargs) File "/opt/scoutsuite/lib/python3.9/site-packages/botocore/paginate.py", line 357, in _make_request return self._method(current_kwargs) File "/opt/scoutsuite/lib/python3.9/site-packages/botocore/client.py", line 534, in _api_call return self._make_api_call(operation_name, kwargs) File "/opt/scoutsuite/lib/python3.9/site-packages/botocore/client.py", line 976, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (Throttling) when calling the DescribeDBParameters operation (reached max retries: 4): Rate exceeded 2023-07-24 12:02:39 ip-10-229-14-28 scout[3328] ERROR base.py L17: Failed to call fetch_all() for resource parameter_groups: 'Parameters' Traceback (most recent call last): File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/base/resources/base.py", line 17, in call await task File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/resources/rds/parametergroups.py", line 14, in fetch_all name, resource = self._parse_parameter_group(raw_parameter_group) File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/resources/rds/parametergroups.py", line 20, in _parse_parameter_group raw_parameter_group['parameters'] = raw_parameter_group.pop('Parameters') KeyError: 'Parameters' 2023-07-24 12:05:08 ip-10-229-14-28 scout[3328] ERROR base.py L17: Failed to call fetch_all() for resource snapshots: 'Attributes' Traceback (most recent call last): File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/base/resources/base.py", line 17, in call await task File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/resources/rds/snapshots.py", line 14, in fetch_all name, resource = self._parse_snapshot(raw_snapshot) File "/opt/scoutsuite/lib/python3.9/site-packages/ScoutSuite/providers/aws/resources/rds/snapshots.py", line 28, in _parse_snapshot snapshot['attributes'] = raw_snapshot['Attributes'] KeyError: 'Attributes'

[GijsbertVanOeveren]

/opt/scoutsuite/bin/scout --version Scout Suite 5.13.0

/opt/scoutsuite/bin/python --version Python 3.9.17

/opt/scoutsuite/bin/python3 --version Python 3.9.17

[GijsbertVanOeveren]

https://github.com/nccgroup/ScoutSuite/wiki/Handling-Rate-Limiting https://docs.aws.amazon.com/AWSEC2/latest/APIReference/throttling.html