nccgroup / Scout2

Security auditing tool for AWS environments
http://nccgroup.github.io/Scout2/
GNU General Public License v2.0
1.73k stars 300 forks source link

error: app crashes after SNS config #196

Closed SkillfulHacking closed 6 years ago

SkillfulHacking commented 6 years ago

cli output: running Python 2.7.10 os x 10.12.1

Crashes every time I run the app with specific account, tested fine with different account with no issue.

Traceback (most recent call last):
  File "/usr/local/bin/Scout2", line 9, in <module>
    load_entry_point('AWSScout2==3.0.5', 'console_scripts', 'Scout2')()
  File "/Library/Python/2.7/site-packages/AWSScout2/__main__.py", line 72, in main
    aws_config['aws_account_id'] = get_aws_account_id(credentials)
  File "/Library/Python/2.7/site-packages/opinel/utils/aws.py", line 98, in get_aws_account_id
    caller_identity = get_caller_identity(credentials)
  File "/Library/Python/2.7/site-packages/opinel/utils/aws.py", line 89, in get_caller_identity
    return api_client.get_caller_identity()
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 253, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 544, in _make_api_call
    operation_model, request_dict)
  File "/Library/Python/2.7/site-packages/botocore/endpoint.py", line 141, in make_request
    return self._send_request(request_dict, operation_model)
  File "/Library/Python/2.7/site-packages/botocore/endpoint.py", line 170, in _send_request
    success_response, exception):
  File "/Library/Python/2.7/site-packages/botocore/endpoint.py", line 249, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/Library/Python/2.7/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/Library/Python/2.7/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL:"https://sts.amazonaws.com/"
SkillfulHacking commented 6 years ago

With debug turned on:

Fetching SNS config...
             regions             topics      subscriptions
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/opinel/utils/aws.py", line 61, in connect_service
    api_client = aws_session.client(**client_params)
  File "/Library/Python/2.7/site-packages/boto3/session.py", line 263, in client
  File "/Library/Python/2.7/site-packages/botocore/session.py", line 836, in create_client
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 63, in create_client
    service_model = self._load_service_model(service_name, api_version)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 93, in _load_service_model
    api_version=api_version)
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 132, in _wrapper
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 374, in load_service_model
    known_services = self.list_available_services(type_name)
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 132, in _wrapper
    data = func(self, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 275, in list_available_services
    d for d in os.listdir(possible_path)
OSError: [Errno 24] Too many open files: '/Library/Python/2.7/site-packages/botocore/data'

Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/AWSScout2/configs/regions.py", line 150, in _fetch_region
    self.regions[region].fetch_all(api_client, self.fetchstatuslogger, params['q'], targets) #  params['targets'])
  File "/Library/Python/2.7/site-packages/AWSScout2/configs/regions.py", line 234, in fetch_all
    self._fetch_targets(api_client, q, target)
  File "/Library/Python/2.7/site-packages/AWSScout2/configs/regions.py", line 240, in _fetch_targets
    list_method = getattr(api_client, list_method_name)
AttributeError: 'NoneType' object has no attribute 'list_topics'

Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/opinel/utils/aws.py", line 61, in connect_service
    api_client = aws_session.client(**client_params)
  File "/Library/Python/2.7/site-packages/boto3/session.py", line 263, in client
  File "/Library/Python/2.7/site-packages/botocore/session.py", line 826, in create_client
  File "/Library/Python/2.7/site-packages/botocore/session.py", line 701, in get_component
    return self._components.get_component(name)
  File "/Library/Python/2.7/site-packages/botocore/session.py", line 897, in get_component
    self._components[name] = factory()
  File "/Library/Python/2.7/site-packages/botocore/session.py", line 186, in create_default_resolver
    endpoints = loader.load_data('endpoints')
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 132, in _wrapper
    data = func(self, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 420, in load_data
    found = self.file_loader.load_file(possible_path)
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 172, in load_file
    with open(full_path, 'rb') as fp:
IOError: [Errno 24] Too many open files: '/Library/Python/2.7/site-packages/botocore/data/endpoints.json'
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/AWSScout2/configs/regions.py", line 150, in _fetch_region
    self.regions[region].fetch_all(api_client, self.fetchstatuslogger, params['q'], targets) #  params['targets'])
  File "/Library/Python/2.7/site-packages/AWSScout2/configs/regions.py", line 234, in fetch_all
    self._fetch_targets(api_client, q, target)
  File "/Library/Python/2.7/site-packages/AWSScout2/configs/regions.py", line 240, in _fetch_targets
    list_method = getattr(api_client, list_method_name)
AttributeError: 'NoneType' object has no attribute 'list_topics'

Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/opinel/utils/aws.py", line 61, in connect_service
    api_client = aws_session.client(**client_params)
  File "/Library/Python/2.7/site-packages/boto3/session.py", line 263, in client
  File "/Library/Python/2.7/site-packages/botocore/session.py", line 836, in create_client
    client_config=config, api_version=api_version)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 63, in create_client
    service_model = self._load_service_model(service_name, api_version)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 93, in _load_service_model
    api_version=api_version)
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 132, in _wrapper
    data = func(self, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 374, in load_service_model
    known_services = self.list_available_services(type_name)
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 132, in _wrapper
    data = func(self, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 279, in list_available_services
    api_versions = os.listdir(full_dirname)
OSError: [Errno 24] Too many open files: '/Library/Python/2.7/site-packages/botocore/data/cur'

Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/AWSScout2/configs/regions.py", line 150, in _fetch_region
    self.regions[region].fetch_all(api_client, self.fetchstatuslogger, params['q'], targets) #  params['targets'])
  File "/Library/Python/2.7/site-packages/AWSScout2/configs/regions.py", line 234, in fetch_all
    self._fetch_targets(api_client, q, target)
  File "/Library/Python/2.7/site-packages/AWSScout2/configs/regions.py", line 240, in _fetch_targets
    list_method = getattr(api_client, list_method_name)
AttributeError: 'NoneType' object has no attribute 'list_topics'

Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/opinel/utils/aws.py", line 61, in connect_service
    api_client = aws_session.client(**client_params)
  File "/Library/Python/2.7/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/Library/Python/2.7/site-packages/botocore/session.py", line 836, in create_client
    client_config=config, api_version=api_version)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 63, in create_client
    service_model = self._load_service_model(service_name, api_version)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 93, in _load_service_model
    api_version=api_version)
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 132, in _wrapper
    data = func(self, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 374, in load_service_model
    known_services = self.list_available_services(type_name)
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 132, in _wrapper
    data = func(self, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 279, in list_available_services
    api_versions = os.listdir(full_dirname)
OSError: [Errno 24] Too many open files: '/Library/Python/2.7/site-packages/botocore/data/datapipeline'
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/opinel/utils/aws.py", line 61, in connect_service
    api_client = aws_session.client(**client_params)
  File "/Library/Python/2.7/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/Library/Python/2.7/site-packages/botocore/session.py", line 836, in create_client
    client_config=config, api_version=api_version)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 63, in create_client
    service_model = self._load_service_model(service_name, api_version)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 93, in _load_service_model
    api_version=api_version)
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 132, in _wrapper
    data = func(self, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 374, in load_service_model
    known_services = self.list_available_services(type_name)
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 132, in _wrapper
    data = func(self, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/botocore/loaders.py", line 279, in list_available_services
    api_versions = os.listdir(full_dirname)
OSError: [Errno 24] Too many open files: '/Library/Python/2.7/site-packages/botocore/data/cognito-identity'

Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/AWSScout2/configs/regions.py", line 150, in _fetch_region
    self.regions[region].fetch_all(api_client, self.fetchstatuslogger, params['q'], targets) #  params['targets'])
  File "/Library/Python/2.7/site-packages/AWSScout2/configs/regions.py", line 234, in fetch_all
    self._fetch_targets(api_client, q, target)
  File "/Library/Python/2.7/site-packages/AWSScout2/configs/regions.py", line 240, in _fetch_targets
    list_method = getattr(api_client, list_method_name)
AttributeError: 'NoneType' object has no attribute 'list_topics'

Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/AWSScout2/configs/regions.py", line 150, in _fetch_region
    self.regions[region].fetch_all(api_client, self.fetchstatuslogger, params['q'], targets) #  params['targets'])
  File "/Library/Python/2.7/site-packages/AWSScout2/configs/regions.py", line 234, in fetch_all
    self._fetch_targets(api_client, q, target)
  File "/Library/Python/2.7/site-packages/AWSScout2/configs/regions.py", line 240, in _fetch_targets
    list_method = getattr(api_client, list_method_name)
AttributeError: 'NoneType' object has no attribute 'list_topics'

                9/14              21/21              18/18
Traceback (most recent call last):
  File "/usr/local/bin/Scout2", line 9, in <module>
    load_entry_point('AWSScout2==3.0.5', 'console_scripts', 'Scout2')()
  File "/Library/Python/2.7/site-packages/AWSScout2/__main__.py", line 72, in main
    aws_config['aws_account_id'] = get_aws_account_id(credentials)
  File "/Library/Python/2.7/site-packages/opinel/utils/aws.py", line 98, in get_aws_account_id
    caller_identity = get_caller_identity(credentials)
  File "/Library/Python/2.7/site-packages/opinel/utils/aws.py", line 89, in get_caller_identity
    return api_client.get_caller_identity()
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 253, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 544, in _make_api_call
    operation_model, request_dict)
  File "/Library/Python/2.7/site-packages/botocore/endpoint.py", line 141, in make_request
    return self._send_request(request_dict, operation_model)
  File "/Library/Python/2.7/site-packages/botocore/endpoint.py", line 170, in _send_request
    success_response, exception):
  File "/Library/Python/2.7/site-packages/botocore/endpoint.py", line 249, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/Library/Python/2.7/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/Library/Python/2.7/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/Library/Python/2.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://sts.amazonaws.com/"
SkillfulHacking commented 6 years ago

I was able to resolve the issue by increasing the number of files that can be open.