Closed huhwhatwhere closed 4 years ago
s3:GetBucketLocation
is in https://github.com/nccgroup/ScoutSuite/wiki/AWS-Minimal-Privileges-Policy. Assuming you've correctly configured the policy with the principal you're running Scout Suite with, the most likely explanation is that those buckets have resource-based policies which disallow you from making that API call against them.
Ok, I can list that bucket from AWS CLI...
I'm also finding that I'm getting these errors
elasticache.py L48: Failed to get ElastiCache security groups: Could not connect to the endpoint URL: "https://elasticache.us-east-1.amazonaws.com/"
future: <Task finished coro=<Trails.fetch_all() done, defined at /Users/xxx/repo/ScoutSuite/ScoutSuite/providers/aws/resources/cloudtrail/trails.py:13> exception=KeyError('EventSelectors')> Traceback (most recent call last): File "/Users/xxrepo/ScoutSuite/ScoutSuite/providers/aws/resources/cloudtrail/trails.py", line 16, in fetch_all name, resource = self._parse_trail(raw_trail) File "/Users/xxx/repo/ScoutSuite/ScoutSuite/providers/aws/resources/cloudtrail/trails.py", line 44, in _parse_trail trail['wildcard_data_logging'] = self.data_logging_status(trail) File "/Users/xxxl/repo/ScoutSuite/ScoutSuite/providers/aws/resources/cloudtrail/trails.py", line 53, in data_logging_status for event_selector in trail['EventSelectors']: KeyError: 'EventSelectors'
I'm actually getting this across majority of AWS services
I've also tried to run this from a container in case I had issues with dependencies, I've also temporarily opened up my IAM policy to FullAdmin
Ok, I can list that bucket from AWS CLI...
For the same principal? There's no reason Scout would fail with the same credentials used for the CLI, as in uses the official library (https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html) under the hood to handle authentication.
I'm actually getting this across majority of AWS services
Please provide full --debug
output.
I've also tried to run this from a container in case I had issues with dependencies, I've also temporarily opened up my IAM policy to FullAdmin
With the same results?
Same result with FullAdmin
Here is the debug outptut
Traceback (most recent call last):
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/connection.py", line 157, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/util/connection.py", line 61, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 752, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/httpsession.py", line 263, in send
chunked=self._chunked(request.headers),
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 376, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
raise value
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
conn.connect()
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/connection.py", line 300, in connect
conn = self._new_conn()
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x117f25290>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/abzmungul/repo/ScoutSuite/ScoutSuite/providers/aws/facade/elasticache.py", line 48, in get_security_groups
'elasticache', region, self.session, 'describe_cache_security_groups', 'CacheSecurityGroups')
File "/Users/abzmungul/repo/ScoutSuite/ScoutSuite/providers/aws/facade/utils.py", line 28, in get_all_pages
service, region, session, paginator_name, [entity], **paginator_args)
File "/Users/abzmungul/repo/ScoutSuite/ScoutSuite/providers/aws/facade/utils.py", line 58, in get_multiple_entities_from_all_pages
return await run_concurrently(lambda: AWSFacadeUtils._get_all_pages_from_paginator(paginator, entities))
File "/Users/abzmungul/repo/ScoutSuite/ScoutSuite/providers/utils.py", line 24, in run_concurrently
return await run_function_concurrently(function)
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/abzmungul/repo/ScoutSuite/ScoutSuite/providers/aws/facade/utils.py", line 58, in <lambda>
return await run_concurrently(lambda: AWSFacadeUtils._get_all_pages_from_paginator(paginator, entities))
File "/Users/abzmungul/repo/ScoutSuite/ScoutSuite/providers/aws/facade/utils.py", line 74, in _get_all_pages_from_paginator
for page in paginator:
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/paginate.py", line 255, in __iter__
response = self._make_request(current_kwargs)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/paginate.py", line 332, in _make_request
return self._method(**current_kwargs)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/client.py", line 613, in _make_api_call
operation_model, request_dict, request_context)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/client.py", line 632, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 137, in _send_request
success_response, exception):
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 231, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 251, in __call__
caught_exception)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 244, in _send
return self.http_session.send(request)
File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/httpsession.py", line 283, in send
raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://elasticache.us-east-2.amazonaws.com/"
Interestingly...i'm running prowler with no issues. Super confused. I've also tried a fresh machine with the same issues.
I've check on AWS orgrnisation just incase there were any SCP policies but nope
Here is the debug outptut
That just looks like network errors, not authentication.
Same result with FullAdmin
Again, if the buckets have resource-based policies then independently of the policies of the principal you're using you might not have access to those buckets.
i'm running prowler with no issues
With the same principal?
At this point all my test are now running as FullAdmin.
I've tested Prowler with the same principal + Full Admin and all work good
Also my issue is not just with S3 buckets, but with majority of the APIs. I'm getting endpoint connection failure on EC2,EMR etc
At this point all my test are now running as FullAdmin.
That makes no difference if the IAM error is caused by resource policies.
I've tested Prowler with the same principal + Full Admin and all work good
That doesn't prove anything, the two tools don't make the exact same API calls.
I'm getting endpoint connection failure on EC2,EMR etc
That has nothing to do with the other issue you mention though, and is most likely a network issue. See https://aws.amazon.com/premiumsupport/knowledge-center/s3-could-not-connect-endpoint-url/.
Hi
I've created an IAM user with the minimal IAM Policy. I keep getting various access denied when running ScoutSuite.
e.g Scout[1133] ERROR s3.py L43: Failed to get bucket location for xxxxxxxxxx-eu-west-1-xxx-xxx: An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied
Running on Any ideas? OSX 10.15.3 Python 3.7.7