(aws-elasticache): Update is failing in stacks containing CfnUserGroup after changing Tags

[deepak-sreekumar]

When using elasticache CfnUserGroup in a stack, the fresh deployment is successful but after changing the Tags in the stack props, the CDK deployment is failing while trying to update the UserGroup. There are no changes made to the UserGroup resource directly.

Found another to this: https://github.com/aws/aws-cdk/issues/15378. It was closed because it was not re-producible.

Reproduction Steps

Consider the stack

import * as cdk from "@aws-cdk/core";
import * as elasticache from "@aws-cdk/aws-elasticache";

export class RedisStack extends cdk.Stack {
    constructor(scope: cdk.Construct, id: string, props: cdk.StackProps) {
        super(scope, id, props);

        const redisDefaultUser = new elasticache.CfnUser(
            this,
            "RedisDefaultUser",
            {
                engine: "redis",
                userId: `redis-default-user`,
                userName: "default",
                passwords: [`dontusethiswaytosetpassword`],
                accessString: "on ~* +@all"
            }
        );

        const redisUserGroup = new elasticache.CfnUserGroup(
            this,
            "RedisUserGroup",
            {
                engine: "redis",
                userGroupId: `redis-user-group-bug`,
                userIds: [redisDefaultUser.ref]
            }
        );
        redisUserGroup.addDependsOn(redisDefaultUser);
    }
}

The stack is created as

import { App } from "@aws-cdk/core";

import { RedisStack } from "./redisStack";

const startCdkDeploy = (): void => {
    //Create the stack
    const app = new App();
    new RedisStack(app, "RedisStack", {
        stackName: "RedisTestStack",
        description: "Redis test stack",
        tags: {
            project: "redis",
            owner: "owner-1"
        }
    });
    app.synth();
};

startCdkDeploy();

The fresh deployment of the stack is successful and upgrade without any changes will also work fine. Then update the stack tag in the constructer to owner-2

tags: {
            project: "redis",
            owner: "owner-2"
        }

Update the stack using cdk deploy

What did you expect to happen?

The stack should get updated without errors and Tags should be updated.

What actually happened?

The stack update is erroring out with the following error.

7:05:32 AM | UPDATE_FAILED        | AWS::ElastiCache::UserGroup | RedisStack/RedisUserGroup
Resource handler returned message: "Please specify the list of user identifiers to be added or removed. (Service: ElastiCache, Status Code: 4
00, Request ID: 5735c315-84ed-4833-99ba-735b0460d46f, Extended Request ID: null)" (RequestToken: da75398a-3185-86af-ccc0-f2e4bef42778, Handle
rErrorCode: GeneralServiceException)

Environment

• *CDK CLI Version :1.116.0 (build d04661d)*
• *Framework Version:1.117.0*
• *Node.js Version:v12.22.4*
• *OS :CentOS Linux release 7.8.2003*
• *Language (Version):TypeScript(3.9.9)*

Other

---

This is :bug: Bug Report

[otaviomacedo]

Thanks for the reproduction steps, @deepak-sreekumar. It allowed to reproduce the issue. Given that all the CDK does in this case is to set the tags when deploying the stack to CloudFormation, this seems like an issue with either CloudFormation or the Elasticache control plane API.

@peterwoodworth, can we follow up internally with these teams?

[peterwoodworth]

Tracking: V432305685

[peterwoodworth]

The internal CFN team has been able to reproduce this issue. They're tracking it internally to be fixed here: V436237202

[brentryan]

Would it be possible to get an update on this issue? Any ETA or workarounds would be much appreciated.

[RafalSladek]

hi there we have the same issue, but there was no change to the stack/cdk, multiple deploys of the same code version lead to this error Do we have any solution or workaround for this?

[jackwarner]

I’d also love to hear an update or discuss any workarounds

[ofiriluz]

any update on the issue?

[peterwoodworth]

I've escalated the issue internally. Been difficult to get an answer - hopefully will have an update soon

[peterwoodworth]

For internal service, code changes are done, and testing needs to take place. Hoping for a release by the end of the week

[peterwoodworth]

well, it looks like it's going to be delayed a few months. I am told by the end of september

[autarchprinceps]

How is CF still treated this badly? This bug has been known for almost a year now, and I just ran into it, so it is STILL not fixed.

[binpatel31]

@peterwoodworth while we are waiting for a fix, is there any workaround to solve this issue without deleting the stack and recreating it?

[ymukhtar]

+1 we are stuck with this issue? any workaround?

[cihansahin]

+1 we are still stuck with this issue. Can we get an update?

[peterwoodworth]

I'm told is supposed to still be on track to release by the end of september, hopefully this will come to fruition

[TylerRockwell]

Any update on a timeline since it's now mid-October?

[peterwoodworth]

The new timeline is by mid-November, thanks for asking @TylerRockwell

[dzavalkin-scayle]

It is already December, obviously no update from AWS and issue is still there...

[peterwoodworth]

I've reached out to service team again, will post an update here when possible

[ThomasSteinbach]

@brentryan @RafalSladek @binpatel31 our workaround was to

• comment out the CfnUserGroup in CDK
• do a deployment (with updated tags)
• comment in the CfnUserGroup and do a further deployment

Despite changing tags with CDK should be super easy, it took us 3 hours to update tags (of our huge stacks) that way. I hope the fix is coming soon. Thre promises for a fix feels like a windows progress bar, which hangs at 99 percent ;)

[peterwoodworth]

Thank you very much for posting a workaround!

I still haven't received any updates or ETA from the team at this point I'm sorry to say, so I wouldn't hold out hope for this coming soon

For amazon employees, the ticket is now being tracked here ELMO-58400

[ThomasSteinbach]

Oh no, it is friday afternoon and I have fallen into exactly the same trap as two months ago 😄 I am glad my workaround still works for me. CDK seems to be a real penetrator for CloudFormation.

I hope Amazon will fix this soon.

[brentryan]

This was resolved last week I believe.

[dmical-icims]

I'm experiencing a similar issue in the eu-central-1 region. On CloudFormation stack update, CloudFormation stack tags are modified. There are no changes being made to the AWS::ElastiCache::UserGroup resource itself in the stack update, but the stack fails to update the resource with the error below. The same stack update succeeds in the us-east-1 and ca-central-1 regions. Is the fix not yet available in the eu-central-1 region?

Resource handler returned message: "Please specify the list of user identifiers to be added or removed. (Service: ElastiCache, Status Code: 400, Request ID: 62f161d0-99dc-4cca-805f-51a1ce02db24, Extended Request ID: null)" (RequestToken: 8e803cff-6052-0b87-2d67-cb7278d569f8, HandlerErrorCode: GeneralServiceException)

[ThomasSteinbach]

Workaround

Since it could take years a solution is in sight, the best temporary solution is not to tag CfnUserGroups. Add this aspect to your stack:

@jsii.implements(aws_cdk.IAspect)
class TagRemoverAspect:
    def visit(self, node):
        # Per default we exclude following resource types from tagging:
        # * Elasticache::UserGroup - has the error: "extraneous key [tags] is not
        if isinstance(node, aws_cdk.aws_elasticache.CfnUserGroup):
            node.add_deletion_override("Properties.Tags")

aws_cdk.Aspects.of(self).add(TagRemoverAspect())

This solution is a more universal approach than excuding tags via the Tags Manager, because you maybe don't know where in your constructs hierarchy you have applied tags.

The aspect will take care, that the CfnUserGroup didn't receive any tags from your code. No tags, no CFN problems.

[peterwoodworth]

This should be fixed except for in eu-central-1 it appears. The team hasn't given any word that they know this is still an issue in that region. Given this is fixed in most regions, and not in our control, I'll let this issue close out if no one comments in the next week

[github-actions[bot]]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.