Cloudformation Stackset Operations seem to be running in sequential mode

faraazkhan commented 2 years ago

We are using the solution as is, with no changes to the defaults for any of the variables defined here: https://github.com/awslabs/aws-control-tower-customizations/blob/main/source/aws/services/cloudformation.py#L28-L35

Yet the operation on stackset seem to only operate at one StackInstance at a time.

> aws cloudformation list-stack-instances --stack-set-name xxxx | grep DetailedStatus 

                "DetailedStatus": "PENDING"
                "DetailedStatus": "PENDING"
                "DetailedStatus": "PENDING"
                "DetailedStatus": "PENDING"
                "DetailedStatus": "RUNNING"
                "DetailedStatus": "PENDING"
                "DetailedStatus": "PENDING"
                "DetailedStatus": "PENDING"
                "DetailedStatus": "PENDING"
                "DetailedStatus": "PENDING"
                "DetailedStatus": "PENDING"
                "DetailedStatus": "PENDING"
                "DetailedStatus": "PENDING"
                "DetailedStatus": "PENDING"
                "DetailedStatus": "PENDING"
                "DetailedStatus": "PENDING"
                "DetailedStatus": "PENDING"

and the FailureTolerancePercentage for operations seems to always be set to zero and RegionOrder is blank i.e. Sequential

{
    "StackSetOperation": {
        "OperationId": "xxxx",
        "StackSetId": "xxxx",
        "Action": "UPDATE",
        "Status": "RUNNING",
        "OperationPreferences": {
            "RegionOrder": [],
            "FailureTolerancePercentage": 0,
            "MaxConcurrentPercentage": 100
        },
        "AdministrationRoleARN": "arn:aws:iam::xxxx:role/service-role/AWSControlTowerStackSetRole",
        "ExecutionRoleName": "AWSControlTowerExecution",
        "CreationTimestamp": "2021-08-26T15:33:09.320000+00:00"
    }
}

To Reproduce

Run the pipeline with one or more stacksets in the manifest.
List Stack Set Instances via the CLI and notice only Stack Set Instance is RUNNING at any given point in time.
Describe operation details for the operation id and notice incorrect value for FailureTolerancePercentage Expected behavior
FailureTolerancePercentage should be set to the default 10
More than one stackset instance should be in RUNNING status
RegionOrder should be set to PARALLEL Please complete the following information about the solution:

[ ] Version: v2.1.0

To get the version of the solution, you can look at the description of the created CloudFormation stack. For example, "(SO0158) - The AWS CloudFormation template for deployment of the Amazon CloudWatch Monitoring Framework. Version v1.0.0". You can also find the version from releases

[ ] Region: us-east-1
[ ] Was the solution modified from the version published on this repository? No
[ ] If the answer to the previous question was yes, are the changes available on GitHub? No
[ ] Have you checked your service quotas for the sevices this solution uses? Yes
[ ] Were there any errors in the CloudWatch Logs? [How to enable debug mode?] No(https://docs.aws.amazon.com/solutions/latest/amazon-cloudwatch-monitoring-framework/troubleshooting.html)

Screenshots If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).

Additional context

groverlalit commented 2 years ago

Ho @faraazkhan Can you please confirm the value provided in CFN parameters and the Lambda Function Environment Variable (FAILED_TOLERANCE_PERCENT) is set to 10 for Fault Tolerance?

faraazkhan commented 2 years ago

This was an issue with the deployment on our side, we have since resolved it, thank you for your support on this.

aws-solutions / aws-control-tower-customizations

Cloudformation Stackset Operations seem to be running in sequential mode #73