StackSet Instances are not being updated

[carlnordenfelt]

Hi,

I am seeing issues with StackSet instances not being updated properly.

I have created a StackSet using the AWS::CloudFormation::StackSet resource. When this resource is changed, for example by changing the value of one of the Parameters, I see that the StackSet is updated and it appears that CloudFormation tries to update the StackInstances but nothing happens happens to them.

I have also tried manually updating the StackSet directly with the same result.

When examining the the UpdateStack event in the target accounts there are no request parameters. This is an excerpt from the CloudTrail event:

{
    "eventVersion": "1.08",
    "userIdentity": {
        "invokedBy": "cloudformation.amazonaws.com"
    },
    "eventTime": "2021-01-04T14:05:26Z",
    "eventSource": "cloudformation.amazonaws.com",
    "eventName": "UpdateStack",
    "errorCode": "ValidationException",
    "errorMessage": "No updates are to be performed.",
    "requestParameters": null,
}

My template, slightly reduced:

StackSet:
        Type: 'AWS::CloudFormation::StackSet'
        Properties:
            AdministrationRoleARN: '....'
            Capabilities:
                - 'CAPABILITY_NAMED_IAM'
            ExecutionRoleName: '....'
            OperationPreferences:
                FailureToleranceCount: 0
                MaxConcurrentCount: 1
            Parameters:
                -   ParameterKey: 'Project'
                     ParameterValue: !Ref 'Project'
                -   ParameterKey: 'Timestamp'
                     ParameterValue: !Ref 'StackSetTimestamp' # This value is changed between updates
            PermissionModel: 'SELF_MANAGED'
            StackInstancesGroup:
                -   DeploymentTargets:
                        Accounts: !Split [ ',', !FindInMap [ 'environment', 'test', 'accounts' ] ]
                    Regions:
                        - 'eu-west-1'
                - !If
                    - 'EnableProdRelease'
                    -   DeploymentTargets:
                            Accounts: !Split [ ',', !FindInMap [ 'environment', 'prod', 'accounts' ] ]
                        Regions:
                            - 'eu-west-1'
                    - !Ref 'AWS::NoValue'
            StackSetName: '....'
            TemplateURL: !Sub 'https://s3-${AWS::Region}.amazonaws.com/...../template.yaml'

Notably, manually updating a StackSet that was created directly in the console works as intended, this appears to only be an issue with the ones created using the CloudFormation resource.

[eduardomourar]

I can confirm the same bug here. Whenever I update any parameter value, it does not trigger the deployment and the parameter details are not actually updated for the stackset itself (checked both through AWS Console and CLI). But if I update the deployment targets or template body, then the deployment of the stack instances get triggered.

[xiwhuang]

Taking a look.

[xiwhuang]

Thanks for reporting this issue, the fix has been merged. Will keep you posted when the fix is live.

[xiwhuang]

Closing the issue as the fix is live , please do not hesitate to contact us if we are still seeing the issue.

[carlnordenfelt]

Hi,

I just tried it out and I still don't see any updates to the Stack Instance in the target account.

[xiwhuang]

Double checking

[xiwhuang]

Hi Carl,

I can confirm that it does work on my end, and my registry type is PUBLIC. In my testing, the StackSets parameter value has been updated once CloudFormation template updates.

Resources:
  TestStackSet:
    Type: AWS::CloudFormation::StackSet
    Properties:
      StackSetName: StackSet-UpdateTest
      Description: Description
      PermissionModel: SERVICE_MANAGED
      AutoDeployment:
        Enabled: true
        RetainStacksOnAccountRemoval: true
      Parameters:
        - ParameterKey: Number
          ParameterValue: 150 ## Update => 300 ##
      TemplateBody: |
        Parameters:
          Number:
            Type: "String"
        Resources:
          WaitCondition: 
            Type: AWS::CloudFormation::WaitCondition
            Properties: 
              Timeout: !Ref Number
Outputs:
  StackSetId:
    Value: !Ref TestStackSet

Are you referring that you still seeing the exact same issue as this issue description?

Many thanks, Xiwei

[carlnordenfelt]

Hi

Yes, same issue. I am running in eu-west-1 if that might make a difference?

To clarify, my StackSet is correctly updated and the parameters are updated like you show. The StackSet Instances in the Target account are not updated.

[xiwhuang]

It sounds like a StackSets service issue. If StackSets gets updated correctly, but it doesn't trigger the stack instances operation.

Question:

1. Are you able to trigger the update of Stack Instances by updating StackSet parameters in StackSets console/API?

2. Would you mind to reach out to Cloud Support to get it to our service team? As it is public repository, I can't get your StackSetId to debug.

[carlnordenfelt]

Hi,

Sorry for the delayed response.

The short answer to your question is no, manual updates do not work either and suffer from the exact same issue.

That said, I want to be very clear about what is and what is not working to avoid confusion:

This issue only affects StackSets created using the AWS::CloudFormation::StackSet Resource. We have plenty of StackSets that were created directly in the Console/CLI and these respond to updates as expected.

[xiwhuang]

Hi Carl,

I just verified we can aslo update the stack instance as expected in eu-west-1.

Screenshots for the Stack Instance

Resources

Parameter has been updated to from Test12345 to Test123456

Testing Template

Resources:
  TestStackSet1:
    Type: AWS::CloudFormation::StackSet
    Properties:
      StackSetName: StackSet-UpdateTest
      Description: Description
      PermissionModel: SELF_MANAGED
      Parameters:
        - ParameterKey: LogGroupName
          ParameterValue: Test123456 # update from Test12345
      StackInstancesGroup:
        - Regions:
            - eu-west-1
          DeploymentTargets:
            Accounts:
              - 316182708337
      TemplateBody: |
        Parameters:
          LogGroupName:
            Type: "String"
        Resources:
          LogGroup: 
            Type: AWS::Logs::LogGroup
            Properties: 
              LogGroupName: !Ref LogGroupName
Outputs:
  StackSetId:
    Value: !Ref TestStackSet1

[carlnordenfelt]

Seems like more research is needed on this, I still see the same behaviour in eu-west-1. Tried updating the StackSet both by changing a Parameter value in the Resource and via manual update via Console. Neither works.

[benbridts]

@carlnordenfelt It might be worth checking if you're having parameter overrides on your StackInstance if you're testing with changing a parameter

[carlnordenfelt]

@benbridts I appreciate the suggestion, it's an easy thing to overlook. Unfortunately that's not the case with our StackSets. No overrides and I have tried updating the template too with the same result.

I think the best indicator or something being wrong are the CT Logs I posted at the very beginning. CloudFormation doesn't appear to get any information about what to update and thus aborts the operation as a no-op:

From the Target account when the parent StackSet is updated:

{
    "eventVersion": "1.08",
    "userIdentity": {
        "invokedBy": "cloudformation.amazonaws.com"
    },
    "eventTime": "2021-01-04T14:05:26Z",
    "eventSource": "cloudformation.amazonaws.com",
    "eventName": "UpdateStack",
    "errorCode": "ValidationException",
    "errorMessage": "No updates are to be performed.",
    "requestParameters": null,
}

[KurtLehnardt]

I just tried updating a parameter to a new value in a stackset and after overriding it with a new value and redeploying via 'edit stackset details', the overridden values did not show up. The parameter I was trying to override is an external ID in an IAM trust policy.