Plumbing never succedes

[gene1wood]

The CloudWatch logs for the GuardDutyEventNormalization lambda function fail 100% of the time.

Grabbing the logs for the last hour and I see this exception on this line of code

An error occurred (UnrecognizedClientException) when calling the PutRule operation: The security token included in the request is invalid: ClientError
Traceback (most recent call last):
  File "/var/task/lambda_functions/plumbing.py", line 185, in handle
    setup_guardduty_plumbing(region_session)
  File "/var/task/lambda_functions/plumbing.py", line 136, in setup_guardduty_plumbing
    Description='Send all guardDuty findings to SNS for SIEM normalization.',
  File "/var/runtime/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the PutRule operation: The security token included in the request is invalid

[sleavitt]

Thought I'd pass along that I think I've discovered the culprit for this (it also affects the invitation_manager.py Lambda function as well).

get_available_regions() returns all regions that a given service is enabled for, including those regions that require opt-in that you may not have opted into as of yet (such as Hong Kong and Bahrain, for instance). The method being called against a region that is not enabled for that account is what is triggering the error message "The security token included in the request is invalid". I think it would be much more helpful if AWS would change the error message under this condition to say something like "The security token included in the request is not valid for region %s" or something like that. But I doubt that's going to happen any time soon.

As a work-around, I created a method called get_enabled_regions() that compares the regions against ec2:DescribeRegions, and used it in place of boto3.get_available_regions(); it seems to work like a charm. The code for it is below, feel free to use as necessary.

def get_enabled_regions(boto_session, service_name):
    enabled_regions = []
    available_regions = boto_session.get_available_regions(service_name)
    client = boto_session.client('ec2')
    response = client.describe_regions(
        Filters=[
            {
                'Name': 'opt-in-status',
                'Values': [
                    'opt-in-not-required',
                    'opted-in'
                ]
            }
        ],
        RegionNames=available_regions,
        AllRegions=False
    )
    for item in response['Regions']:
        enabled_regions.extend([item['RegionName']])
    return enabled_regions