search

awslabs / aws-securityhub-multiaccount-scripts

This script automates the process of running the Security Hub multi-account workflow across a group of accounts that are in your control
MIT No Attribution
271 stars 108 forks source link

Fails and leaves the accounts in a broken state... #53

Open max-allan-surevine opened 4 years ago

max-allan-surevine commented 4 years ago

Ran the script and it failed quite often.

MaAl00350:aws-securityhub-multiaccount-scripts max [master] $ ./enablesecurityhub.py  --master_account 161606123770 --assume_role fromCore org.csv --enabled_regions eu-west-1,eu-west-2
WARNING: Executing a script that is loading libcrypto in an unsafe way. This will fail in a future version of macOS. Set the LIBRESSL_REDIRECT_STUB_ABORT=1 in the environment to force this into an error.
Enabling members in these regions: ['eu-west-1', 'eu-west-2']
Assumed session for 161606123770.
Assumed session for 177825663049.
Beginning 177825663049 in eu-west-1
Error InsufficientDeliveryPolicyException(u'An error occurred (InsufficientDeliveryPolicyException) when calling the PutDeliveryChannel operation: Insufficient delivery policy to s3 bucket: config-bucket-177825663049, unable to assume role: arn:aws:iam::177825663049:role/aws-service-role/config.amazonaws.com/AWSServiceRoleForConfig.',) enabling Config on account 177825663049
Added Account 177825663049 to member list in SecurityHub master account 161606123770 for region eu-west-1
Invited Account 177825663049 to SecurityHub master account 161606123770 in region eu-west-1
Accepting Account 177825663049 to SecurityHub master account 161606123770 in region eu-west-1
Finished 177825663049 in eu-west-1
Beginning 177825663049 in eu-west-2
Added Account 177825663049 to member list in SecurityHub master account 161606123770 for region eu-west-2
Invited Account 177825663049 to SecurityHub master account 161606123770 in region eu-west-2
Accepting Account 177825663049 to SecurityHub master account 161606123770 in region eu-west-2
Finished 177825663049 in eu-west-2
Assumed session for 304071828426.
Beginning 304071828426 in eu-west-1
Added Account 304071828426 to member list in SecurityHub master account 161606123770 for region eu-west-1
Invited Account 304071828426 to SecurityHub master account 161606123770 in region eu-west-1
Accepting Account 304071828426 to SecurityHub master account 161606123770 in region eu-west-1
Finished 304071828426 in eu-west-1
Beginning 304071828426 in eu-west-2
Added Account 304071828426 to member list in SecurityHub master account 161606123770 for region eu-west-2
Invited Account 304071828426 to SecurityHub master account 161606123770 in region eu-west-2
Accepting Account 304071828426 to SecurityHub master account 161606123770 in region eu-west-2
Finished 304071828426 in eu-west-2
Assumed session for 417831697585.
Beginning 417831697585 in eu-west-1
Added Account 417831697585 to member list in SecurityHub master account 161606123770 for region eu-west-1
Invited Account 417831697585 to SecurityHub master account 161606123770 in region eu-west-1
Accepting Account 417831697585 to SecurityHub master account 161606123770 in region eu-west-1
Finished 417831697585 in eu-west-1
Beginning 417831697585 in eu-west-2
Added Account 417831697585 to member list in SecurityHub master account 161606123770 for region eu-west-2
Invited Account 417831697585 to SecurityHub master account 161606123770 in region eu-west-2
Accepting Account 417831697585 to SecurityHub master account 161606123770 in region eu-west-2
Finished 417831697585 in eu-west-2
Assumed session for 086867758037.
Beginning 086867758037 in eu-west-1
Error InsufficientDeliveryPolicyException(u'An error occurred (InsufficientDeliveryPolicyException) when calling the PutDeliveryChannel operation: Insufficient delivery policy to s3 bucket: config-bucket-086867758037, unable to assume role: arn:aws:iam::086867758037:role/aws-service-role/config.amazonaws.com/AWSServiceRoleForConfig.',) enabling Config on account 086867758037
Added Account 086867758037 to member list in SecurityHub master account 161606123770 for region eu-west-1
Invited Account 086867758037 to SecurityHub master account 161606123770 in region eu-west-1
Accepting Account 086867758037 to SecurityHub master account 161606123770 in region eu-west-1
Finished 086867758037 in eu-west-1
Beginning 086867758037 in eu-west-2
Added Account 086867758037 to member list in SecurityHub master account 161606123770 for region eu-west-2
Invited Account 086867758037 to SecurityHub master account 161606123770 in region eu-west-2
Accepting Account 086867758037 to SecurityHub master account 161606123770 in region eu-west-2
Finished 086867758037 in eu-west-2
Assumed session for 083816131855.
Beginning 083816131855 in eu-west-1
Error InsufficientDeliveryPolicyException(u'An error occurred (InsufficientDeliveryPolicyException) when calling the PutDeliveryChannel operation: Insufficient delivery policy to s3 bucket: config-bucket-083816131855, unable to assume role: arn:aws:iam::083816131855:role/aws-service-role/config.amazonaws.com/AWSServiceRoleForConfig.',) enabling Config on account 083816131855
Added Account 083816131855 to member list in SecurityHub master account 161606123770 for region eu-west-1
Invited Account 083816131855 to SecurityHub master account 161606123770 in region eu-west-1
Accepting Account 083816131855 to SecurityHub master account 161606123770 in region eu-west-1
Finished 083816131855 in eu-west-1
Beginning 083816131855 in eu-west-2
Added Account 083816131855 to member list in SecurityHub master account 161606123770 for region eu-west-2
Invited Account 083816131855 to SecurityHub master account 161606123770 in region eu-west-2
Accepting Account 083816131855 to SecurityHub master account 161606123770 in region eu-west-2
Finished 083816131855 in eu-west-2
Assumed session for 401787195176.
Beginning 401787195176 in eu-west-1
Error InsufficientDeliveryPolicyException(u'An error occurred (InsufficientDeliveryPolicyException) when calling the PutDeliveryChannel operation: Insufficient delivery policy to s3 bucket: config-bucket-401787195176, unable to assume role: arn:aws:iam::401787195176:role/aws-service-role/config.amazonaws.com/AWSServiceRoleForConfig.',) enabling Config on account 401787195176
Added Account 401787195176 to member list in SecurityHub master account 161606123770 for region eu-west-1
Invited Account 401787195176 to SecurityHub master account 161606123770 in region eu-west-1
Accepting Account 401787195176 to SecurityHub master account 161606123770 in region eu-west-1
Finished 401787195176 in eu-west-1
Beginning 401787195176 in eu-west-2
Added Account 401787195176 to member list in SecurityHub master account 161606123770 for region eu-west-2
Invited Account 401787195176 to SecurityHub master account 161606123770 in region eu-west-2
Accepting Account 401787195176 to SecurityHub master account 161606123770 in region eu-west-2
Finished 401787195176 in eu-west-2

......
---------------------------------------------------------------
Failed Accounts
---------------------------------------------------------------
177825663049: 
    Error validating or enabling AWS Config for account 177825663049 in eu-west-1 - requested standards not enabled
086867758037: 
    Error validating or enabling AWS Config for account 086867758037 in eu-west-1 - requested standards not enabled
083816131855: 
    Error validating or enabling AWS Config for account 083816131855 in eu-west-1 - requested standards not enabled
401787195176: 
    Error validating or enabling AWS Config for account 401787195176 in eu-west-1 - requested standards not enabled
486105608128: 
    Error validating or enabling AWS Config for account 486105608128 in eu-west-1 - requested standards not enabled

My role has the built in AWS "AdministratorAccess" policy in the 083816131855 account.

When I try to enable Config by hand in eu-west-1 in the console, I get an error :

AWS Config cannot start recording because the delivery channel was not found.

In eu-west-2 it has created a delivery channel but not in eu-west-1 :

MaAl:aws-securityhub-multiaccount-scripts max [master] $ aws configservice describe-delivery-channels --region eu-west-2
{
    "DeliveryChannels": [
        {
            "name": "config-s3-delivery",
            "s3BucketName": "config-bucket-083816131855",
            "configSnapshotDeliveryProperties": {
                "deliveryFrequency": "TwentyFour_Hours"
            }
        }
    ]
}
MaAl:aws-securityhub-multiaccount-scripts max [master] $ aws configservice describe-delivery-channels --region eu-west-1
{
    "DeliveryChannels": []
}

If I take the DeliveryChannel json from eu-west-2, I can apply it to eu-west-1 with a put-delivery-channel CLI command.

And then enable config from the console.

I believe the cause of the problem is that you are not waiting for the AWSServiceRoleForConfig to be fully created before using it. IAM is a global service and it takes time for changes to replicate around the globe.

https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_general.html

We recommend that you do not include such IAM changes in the critical, high-availability code paths of your application. Instead, make IAM changes in a separate initialization or setup routine that you run less frequently. Also, be sure to verify that the changes have been propagated before production workflows depend on them.

A loop through all accounts creating the role first and then doing the work would be a more reliable design.

Rerunning the script seems to be fixing it.