It fails with incorrect error for each region not already enabled

[dfuentes77]

I run the script, with the list of regions and arns for standards and it enables securityhub for the first region in the list and then exits the script with the error: Error: Unable to enable Security Hub on Master account in region us-west-1

But when i go to AWS console, that region's SecurityHub was enabled

If i disable securityhub for that region and run the script again, it'll display the same thing, but enable it again

If I leave the region enabled and run the script again, it'll enable the next region in the list, but exit with the same error for that region:

[ec2-user@<host> aws-securityhub-multiaccount-scripts]$ ./enablesecurityhub.py --master_account <account-id> --enable_standards arn:aws:securityhub:::ruleset/cis-aws-foundations-benchmark/v/1.2.0,arn:aws:securityhub:us-west-2::standards/aws-foundational-security-best-practices/v/1.0.0 --assume_role ManageSecurityHub --enabled_regions us-west-1,us-west-2,us-east-1,us-east-2,eu-west-1,eu-west-2,eu-west-3,eu-central-1,eu-north-1,ap-south-1,ap-northeast-1,ap-northeast-2,ap-northeast-3,ap-southeast-1,ap-southeast-2,ca-central-1,sa-east-1 members.csv

        *****************************************************************************************************************************************************************************************
        *      By turning on this Standards you will enable security evaluations to run. For current pricing and example scenarios please refer to the current AWS Security Hub pricing.        *
        *      Important: You must enable AWS Config for all resources in each AWS Region where you will be running a Standard. If Config is not already enabled it will be enabled and         *
        *      configured in each region.                                                                                                                                                       *
        *                                                                                                                                                                                       *
        *      In addition to AWS Security Hub charges, you will also incur charges for the Configuration Items recorded by AWS Config, as per the AWS Config pricing. These charges are        *
        *      separate from (and not included in) AWS Security Hub pricing.                                                                                                                    *
        *****************************************************************************************************************************************************************************************

        Continue?(yes/no):

yes
Enabling members in these regions: ['us-west-1', 'us-west-2', 'us-east-1', 'us-east-2', 'eu-west-1', 'eu-west-2', 'eu-west-3', 'eu-central-1', 'eu-north-1', 'ap-south-1', 'ap-northeast-1', 'ap-northeast-2', 'ap-northeast-3', 'ap-southeast-1', 'ap-southeast-2', 'ca-central-1', 'sa-east-1']
Enabling the following Security Hub Standards for enabled account(s) and region(s): ['arn:aws:securityhub:::ruleset/cis-aws-foundations-benchmark/v/1.2.0', 'arn:aws:securityhub:us-west-2::standards/aws-foundational-security-best-practices/v/1.0.0']
Assumed session for <account-id>.
Error: Unable to enable Security Hub on Master account in region us-west-1

[ryanholland]

The format for standards other than CIS is to not include the full ARN but rather the standard name, ie: --enable_standards arn:aws:securityhub:::ruleset/cis-aws-foundations-benchmark/v/1.2.0,standards/aws-foundational-security-best-practices/v/1.0.0

can you try with that change

[dfuentes77]

Thanks for that tip. That seemed to take me further except now I'm running into a new error. It seems to successfully enable the standards and setup the master/member relationships for some regions but after awhile it hits this:

Beginning 269xxxxxxx in eu-west-1
Error Processing Account 269xxxxxxx
---------------------------------------------------------------
Failed Accounts
---------------------------------------------------------------
269xxxxxxx:
    Error validating or enabling AWS Config for account 269xxxxxxx in us-west-1 - requested standards not enabled
269xxxxxxx:
    ClientError(u'An error occurred (AccessDeniedException) when calling the DescribeConfigurationRecorders operation: User: arn:aws:sts::269xxxxxxx:assumed-role/ManageSecurityHub/EnableSecurityHub is not authorized to perform: config:DescribeConfigurationRecorders with an explicit deny',)

I ran it again without changing anything, hoping that it was idempotent, and it confirmed the standards and member relationships were already enabled, but failed again with:

Beginning 269xxxxxxx in eu-west-1
Error Processing Account 269xxxxxxx
---------------------------------------------------------------
Failed Accounts
---------------------------------------------------------------
269xxxxxxx:
    ClientError(u'An error occurred (AccessDeniedException) when calling the DescribeConfigurationRecorders operation: User: arn:aws:sts::269xxxxxxx:assumed-role/ManageSecurityHub/EnableSecurityHub is not authorized to perform: config:DescribeConfigurationRecorders with an explicit deny',)
---------------------------------------------------------------

And as suggested, the appropriate cloudformation template was used to add the specified role to the member account. I also confirmed that this permission does seem to be specified in that role.

So I'm unclear why it's complaining.

[ryanholland]

The error references an explicit deny which would take precedence over the allow in the roles policy, do you perhaps have a service control policy that with an explicit deny?

[dfuentes77]

The role is as specified by the cloudformation template. There is no other policy added to that role

[ryanholland]

Service Control Policies are set at the AWS Organization level and supersede any permissions that are granted in the role policies. Do you have any SCPs enabled? From the AccessDenied error above, something has explicitly denied the request, which means a policy has a Deny statement in it that is blocking the allow statement in the role policy.

[dfuentes77]

Thanks for pointing that out. I forgot about that context. I do have an SCP that restricts regions to just US regions. As this applies only for member accounts in Organizations, how do I make that distinction here? I want to master account to have security hub turned on for all regions but only the limited (US) regions turned on and the master/member relationship setup in the member accounts.

[dfuentes77]

Any help on this would be appreciated. Is this just something the script doesn't support? Is there a work around or do I need to actually turn off the region restriction I have in the SCP?

[ryanholland]

You can use the --enabled_regions to limit the enablement to the regions you are not blocking, now that you've ran the script previously without that it should have already enabled Security Hub in all regions at the master level. That said the errors you received show the failed regions, it doesn't stop processing at that point but rather continues to try and will succeed in regions you're not blocking. So you should have the service enabled and linked in all the regions you are not blocking at this point.

[dfuentes77]

I actually did use --enabled_regions in the first command above. The list just included all enabled regions in the master account. I do see SecurityHub enabled in all master account regions and the master/member seems setup for the US regions, but I'm not seeing AWS Config enabled as stated in the documentation (and in the code) in all regions (because the scripts stops). So many of the checks aren't working since SecurityHub uses Config Rules. The console is also showing the error:

So if I reduce that --enabled_region list to just US regions, it won't enable AWS Config for all other regions in the master account, correct?

[dfuentes77]

Thanks for the help Ryan. What do you suggest at this point? Do I need to run it with the just the US regions and then I have to enable AWS Config on other Master account regions manually? Do I also have to enable all the SecurityHub Config Rules that it needs, in each region manually?

[ryanholland]

Are you sure that the master does not have config enabled, is it possible that the related findings are for the member accounts? When you ran the script w/o --enabled_regions it should have enabled Sec Hub and Config in the master account in all regions.

[dfuentes77]

I tried but couldn't run it w/o --enabled_regions initially since we have 4 regions that are not enabled by default and therefore disabled and it would fail.

[dfuentes77]

I just tried running it with US regions only:

./enablesecurityhub.py --master_account <master id> --enable_standards arn:aws:securityhub:::ruleset/cis-aws-foundations-benchmark/v/1.2.0,standards/aws-foundational-security-best-practices/v/1.0.0 --assume_role ManageSecurityHub --enabled_regions us-west-1,us-west-2,us-east-1,us-east-2 members.csv

and this was output:

        Continue?(yes/no):

yes
Enabling members in these regions: ['us-west-1', 'us-west-2', 'us-east-1', 'us-east-2']
Enabling the following Security Hub Standards for enabled account(s) and region(s): ['arn:aws:securityhub:::ruleset/cis-aws-foundations-benchmark/v/1.2.0', 'standards/aws-foundational-security-best-practices/v/1.0.0']
Assumed session for <master id>.
Assumed session for <member id>.
Beginning <member id> in us-west-1
Finished enabling stanard arn:aws:securityhub:::ruleset/cis-aws-foundations-benchmark/v/1.2.0 on account <member id> for region us-west-1
Finished enabling stanard arn:aws:securityhub:us-west-1::standards/aws-foundational-security-best-practices/v/1.0.0 on account <member id> for region us-west-1
Account <member id> is already a member of <master id> in region us-west-1
Account <member id> is already enabled
Beginning <member id> in us-west-2
Finished enabling stanard arn:aws:securityhub:::ruleset/cis-aws-foundations-benchmark/v/1.2.0 on account <member id> for region us-west-2
Finished enabling stanard arn:aws:securityhub:us-west-2::standards/aws-foundational-security-best-practices/v/1.0.0 on account <member id> for region us-west-2
Account <member id> is already a member of <master id> in region us-west-2
Account <member id> is already enabled
Beginning <member id> in us-east-1
Finished enabling stanard arn:aws:securityhub:::ruleset/cis-aws-foundations-benchmark/v/1.2.0 on account <member id> for region us-east-1
Finished enabling stanard arn:aws:securityhub:us-east-1::standards/aws-foundational-security-best-practices/v/1.0.0 on account <member id> for region us-east-1
Account <member id> is already a member of <master id> in region us-east-1
Account <member id> is already enabled
Beginning <member id> in us-east-2
Finished enabling stanard arn:aws:securityhub:::ruleset/cis-aws-foundations-benchmark/v/1.2.0 on account <member id> for region us-east-2
Finished enabling stanard arn:aws:securityhub:us-east-2::standards/aws-foundational-security-best-practices/v/1.0.0 on account <member id> for region us-east-2
Account <member id> is already a member of <master id> in region us-east-2
Account <member id> is already enabled

and completes without error, but AWS Config recorder has NOT been auto configured for any of the other regions. Oregon's AWS Config was previously configured because that is our primary region.

[dfuentes77]

One thing I noticed when trying to find out what's happening, is that it has the us-east-1 region hard-coded for the S3 bucket. Our existing config bucket already exists in another region. https://github.com/awslabs/aws-securityhub-multiaccount-scripts/blob/master/enablesecurityhub.py#L101

[dfuentes77]

So I tried setting the region for our existing config s3 bucket manually in the script and running the script again. It didn't seem to help. It's not throwing any errors when attempting to turn on AWS Config Recorder in the regions, but it never actually does it. I've attempted to turn on AWS Config manually in one of those regions but then I noticed I have to then add the SecurityHub Config rules manually. I'm guessing that happens automatically when SecurityHub is enabled and AWS Config Recorder is already enabled. So I now have to disable SecurityHub everywhere. I tried the disable script but when attempting with default all regions or restricting to the long list of regions that are enabled in our AWS master account, it won't disable SecurityHub in all those regions.

[dfuentes77]

So now I had to disable SecurityHub in each of those regions manually in the AWS Console, enable AWS Config Recorder in each of the regions where it's not already enabled manually in the console, and then re-enable SecurityHub in each of the regions in the master account manually. I then used the enable script to automate the setting up of master/member for SecurityHub for only the US regions that the member account is restricted to (by SCP).