Access Denied

[huhwhatwhere]

Hi

I've created an IAM user with the minimal IAM Policy. I keep getting various access denied when running ScoutSuite.

e.g Scout[1133] ERROR s3.py L43: Failed to get bucket location for xxxxxxxxxx-eu-west-1-xxx-xxx: An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied

Running on Any ideas? OSX 10.15.3 Python 3.7.7

[x4v13r64]

s3:GetBucketLocation is in https://github.com/nccgroup/ScoutSuite/wiki/AWS-Minimal-Privileges-Policy. Assuming you've correctly configured the policy with the principal you're running Scout Suite with, the most likely explanation is that those buckets have resource-based policies which disallow you from making that API call against them.

[huhwhatwhere]

Ok, I can list that bucket from AWS CLI...

I'm also finding that I'm getting these errors

elasticache.py L48: Failed to get ElastiCache security groups: Could not connect to the endpoint URL: "https://elasticache.us-east-1.amazonaws.com/"

future: <Task finished coro=<Trails.fetch_all() done, defined at /Users/xxx/repo/ScoutSuite/ScoutSuite/providers/aws/resources/cloudtrail/trails.py:13> exception=KeyError('EventSelectors')> Traceback (most recent call last): File "/Users/xxrepo/ScoutSuite/ScoutSuite/providers/aws/resources/cloudtrail/trails.py", line 16, in fetch_all name, resource = self._parse_trail(raw_trail) File "/Users/xxx/repo/ScoutSuite/ScoutSuite/providers/aws/resources/cloudtrail/trails.py", line 44, in _parse_trail trail['wildcard_data_logging'] = self.data_logging_status(trail) File "/Users/xxxl/repo/ScoutSuite/ScoutSuite/providers/aws/resources/cloudtrail/trails.py", line 53, in data_logging_status for event_selector in trail['EventSelectors']: KeyError: 'EventSelectors'

I'm actually getting this across majority of AWS services

I've also tried to run this from a container in case I had issues with dependencies, I've also temporarily opened up my IAM policy to FullAdmin

[x4v13r64]

Ok, I can list that bucket from AWS CLI...

For the same principal? There's no reason Scout would fail with the same credentials used for the CLI, as in uses the official library (https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html) under the hood to handle authentication.

I'm actually getting this across majority of AWS services

Please provide full --debug output.

I've also tried to run this from a container in case I had issues with dependencies, I've also temporarily opened up my IAM policy to FullAdmin

With the same results?

[huhwhatwhere]

Same result with FullAdmin

Here is the debug outptut

Traceback (most recent call last):
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/connection.py", line 157, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 752, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/httpsession.py", line 263, in send
    chunked=self._chunked(request.headers),
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 376, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/connection.py", line 300, in connect
    conn = self._new_conn()
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x117f25290>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/abzmungul/repo/ScoutSuite/ScoutSuite/providers/aws/facade/elasticache.py", line 48, in get_security_groups
    'elasticache', region, self.session, 'describe_cache_security_groups', 'CacheSecurityGroups')
  File "/Users/abzmungul/repo/ScoutSuite/ScoutSuite/providers/aws/facade/utils.py", line 28, in get_all_pages
    service, region, session, paginator_name, [entity], **paginator_args)
  File "/Users/abzmungul/repo/ScoutSuite/ScoutSuite/providers/aws/facade/utils.py", line 58, in get_multiple_entities_from_all_pages
    return await run_concurrently(lambda: AWSFacadeUtils._get_all_pages_from_paginator(paginator, entities))
  File "/Users/abzmungul/repo/ScoutSuite/ScoutSuite/providers/utils.py", line 24, in run_concurrently
    return await run_function_concurrently(function)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/abzmungul/repo/ScoutSuite/ScoutSuite/providers/aws/facade/utils.py", line 58, in <lambda>
    return await run_concurrently(lambda: AWSFacadeUtils._get_all_pages_from_paginator(paginator, entities))
  File "/Users/abzmungul/repo/ScoutSuite/ScoutSuite/providers/aws/facade/utils.py", line 74, in _get_all_pages_from_paginator
    for page in paginator:
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/paginate.py", line 255, in __iter__
    response = self._make_request(current_kwargs)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/paginate.py", line 332, in _make_request
    return self._method(**current_kwargs)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/client.py", line 613, in _make_api_call
    operation_model, request_dict, request_context)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/client.py", line 632, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 137, in _send_request
    success_response, exception):
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 231, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/endpoint.py", line 244, in _send
    return self.http_session.send(request)
  File "/Users/abzmungul/repo/ScoutSuite/venv/lib/python3.7/site-packages/botocore/httpsession.py", line 283, in send
    raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://elasticache.us-east-2.amazonaws.com/"

[huhwhatwhere]

Interestingly...i'm running prowler with no issues. Super confused. I've also tried a fresh machine with the same issues.

I've check on AWS orgrnisation just incase there were any SCP policies but nope

[x4v13r64]

Here is the debug outptut

That just looks like network errors, not authentication.

Same result with FullAdmin

Again, if the buckets have resource-based policies then independently of the policies of the principal you're using you might not have access to those buckets.

i'm running prowler with no issues

With the same principal?

[huhwhatwhere]

At this point all my test are now running as FullAdmin.

I've tested Prowler with the same principal + Full Admin and all work good

Also my issue is not just with S3 buckets, but with majority of the APIs. I'm getting endpoint connection failure on EC2,EMR etc

[x4v13r64]

At this point all my test are now running as FullAdmin.

That makes no difference if the IAM error is caused by resource policies.

I've tested Prowler with the same principal + Full Admin and all work good

That doesn't prove anything, the two tools don't make the exact same API calls.

I'm getting endpoint connection failure on EC2,EMR etc

That has nothing to do with the other issue you mention though, and is most likely a network issue. See https://aws.amazon.com/premiumsupport/knowledge-center/s3-could-not-connect-endpoint-url/.