search

aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code
https://aws.amazon.com/cdk
Apache License 2.0
11.61k stars 3.91k forks source link

CLI: Hotswap - support `ConfigurationEndpoint.Address` attribute of the `AWS::ElastiCache::CacheCluster` resource #18809

Open automartin5000 opened 2 years ago

automartin5000 commented 2 years ago

What is the problem?

Can't use cdk deploy --hotswap with an ElastiCache CacheCluster. Get the error:

Error: We don't support attributes of the 'AWS::ElastiCache::CacheCluster' resource. 
This is a CDK limitation. Please report it at https://github.com/aws/aws-cdk/issues/new/choose

Just following the instructions in the error :)

Reproduction Steps

Try to run cdk deploy --hotswap on a previously deployed cdk stack with a Fargate Service/Task that has an ElastiCache endpoint in it's environment variables.

What did you expect to happen?

Successful hotswap

What actually happened?

Received an error

Error: We don't support attributes of the 'AWS::ElastiCache::CacheCluster' resource. 
This is a CDK limitation. Please report it at https://github.com/aws/aws-cdk/issues/new/choose

CDK CLI Version

2.8.0

Framework Version

No response

Node.js Version

v16.13.2

OS

Mac OS 12.1

Language

Python

Language Version

No response

Other information

No response

skinny85 commented 2 years ago

Hi @automartin5000,

thanks for opening the issue! It's great that the instructions in the error message did its job 😃.

Can you show a little bit of your CDK code that sets those environment variables? That will make it easier for me to reproduce 🙂.

Thanks, Adam

automartin5000 commented 2 years ago

Hey @skinny85,

Is this helpful: self.scope.app_env_vars["CACHE_LOCATION"] = self.memcache_cluster.attr_configuration_endpoint_address

skinny85 commented 2 years ago

Yes, but if you could show me how self. memcache_cluster is created, that would be helpful too 🙂.

automartin5000 commented 2 years ago

@skinny85

Here you go, let me know if you need more.


        self.memcache_cluster = elasticache.CfnCacheCluster(
            self,
            "ElastiCacheCluster",
            **elasticache_params,
            engine="memcached",
            vpc_security_group_ids=[
                self.elasticache_security_group.get_att("GroupId").to_string()
            ],
            cache_subnet_group_name=self.elasticache_subnet_group.ref
        )
        self.scope.app_env_vars["CACHE_LOCATION"] = self.memcache_cluster.attr_configuration_endpoint_address
skinny85 commented 2 years ago

Almost! 😄 What's self.scope.app_env_vars? Is it just a simple dictionary, or something else?

automartin5000 commented 2 years ago

Yeah just a dictionary that gets passed into the environment arg for the task definition

skinny85 commented 2 years ago

Sounds good! Thanks for all of the info. I know what I need to do 😉.

skinny85 commented 2 years ago

Actually, I may have spoken too soon 😕.

The problem is that you're using an attribute of the AWS::ElastiCache::CacheCluster resource (ConfigurationEndpoint.Address) that can't be easily constructed from the name of the Cluster, because it's a URL that's generated by the ElastiCache Service, like tes-ca-10pps2g12dfv.zy7new.cfg.usw2.cache.amazonaws.com. That means it's really difficult to get the value of that attribute when hotswapping - we would have to do an ElastiCache-specific service call to do it, and we don't have a mechanism for this now.

Apologies for the bad experience @automartin5000, but it won't be easy to support hotswapping for this particular use case 😕.

I'll keep this issue open to track this gap, but it will be non-trivial to fix for sure.

automartin5000 commented 2 years ago

Thanks @skinny85 for looking into this so quickly. What about a flag to just ignore unsupported CFN attributes and echo a warning on deploy?

skinny85 commented 2 years ago

Isn't that basically what happens now though? (I understand that the output says "Error", but in fact the deployment should proceed, just not a hotswap one - a full CloudFormation one. Is this not what happens to you?)

Or are you saying that you want a flag to suppress showing the warning about AWS::ElastiCache::CacheCluster?

automartin5000 commented 2 years ago
Screen Shot 2022-02-04 at 21 22 58

Does not continue, just exits ^^

skinny85 commented 2 years ago

Hmmm. That's not what I expected 😛.

What command did you execute here? Seeing the entire console output, preferably with the --verbose switch, would be super helpful (feel free to use GitHub Gists if it's too long).

automartin5000 commented 2 years ago

I'd rather not dump the entire output here with IAM role names, resource IDs, and account numbers. The command I ran was: cdk deploy --hotswap dev --app 'cdk.out/' --require-approval never

skinny85 commented 2 years ago

Hmmm. I've just tried it on my own Stack, very simple:

class CdkWatchMinimalEcsServiceStack extends cdk.Stack {
    constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
        super(scope, id, props);

        // Fargate does not work with default VPCs
        const vpc = new ec2.Vpc(this, 'Vpc', {
            maxAzs: 2, // ALB requires 2 AZs
            natGateways: 1,
        });
        const defaultVpc = ec2.Vpc.fromLookup(this, 'DefaultVpc', { isDefault: true });
        const cacheCluster = new elasticache.CfnCacheCluster(this, 'ElasticacheCluster', {
            vpcSecurityGroupIds: [new ec2.SecurityGroup(this, 'ElasticacheSecurityGroup', {
                vpc: defaultVpc,
            }).securityGroupId],
            numCacheNodes: 1,
            cacheNodeType: 'cache.m6g.large',
            engine: 'memcached',
        });
        const service = new ecs_patterns.ApplicationLoadBalancedFargateService(this, 'EcsService', {
            vpc,
            taskImageOptions: {
                image: ecs.ContainerImage.fromAsset('image-asset'),
                containerPort: 3000,
                environment: {
                    'CACHE_CLUSTER': cacheCluster.attrConfigurationEndpointAddress,
                },
            },
        });
    }
}

And this is what I see:

$ yarn cdk deploy --hotswap

✨  Synthesis time: 2.62s

⚠️ The --hotswap flag deliberately introduces CloudFormation drift to speed up deployments
⚠️ It should only be used for development - never use it for your production Stacks!
CdkWatchMinimalEcsServiceStack: deploying...
[0%] start: Publishing 34491e4fbe3368c8f6ffc4027d21a5241cbea4412bf22ae9af7dfe87255b019a:current
[100%] success: Published 34491e4fbe3368c8f6ffc4027d21a5241cbea4412bf22ae9af7dfe87255b019a:current
Could not perform a hotswap deployment, because the CloudFormation template could not be resolved: We don't support attributes of the 'AWS::ElastiCache::CacheCluster' resource. This is a CDK limitation. Please report it at https://github.com/aws/aws-cdk/issues/new/choose
Falling back to doing a full deployment
CdkWatchMinimalEcsServiceStack: creating CloudFormation changeset...
[██████████████▌···········································] (1/4)

4:26:00 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack                | CdkWatchMinimalEcsServiceStack
4:26:49 PM | UPDATE_IN_PROGRESS   | AWS::ECS::Service                         | EcsService/Service/Service

A few more questions, @automartin5000, so we can dig more into this:

  1. What version of the CLI are you using? (You can check that with cdk --version)
  2. Why are you running cdk deploy --hotswap --app 'cdk.out'? Why not just cdk deploy --hotswap? Doing it with the --app parameter means your changes will not be picked up unless you explicitly call cdk synth, so I wonder why introduce this extra step (cdk deploy --hotswap will call synth for you, automatically).
skinny85 commented 2 years ago

I tried it with --app cdk.out too, just to make sure I wasn't going crazy - same result:

$ yarn cdk deploy --hotswap --app cdk.out

✨  Synthesis time: 0.02s

⚠️ The --hotswap flag deliberately introduces CloudFormation drift to speed up deployments
⚠️ It should only be used for development - never use it for your production Stacks!
CdkWatchMinimalEcsServiceStack: deploying...
[0%] start: Publishing 06134eb41d4fad8134760d5ee6f5830f70d1e35384186ac44622b3d251fe3b20:current
[100%] success: Published 06134eb41d4fad8134760d5ee6f5830f70d1e35384186ac44622b3d251fe3b20:current
Could not perform a hotswap deployment, because the CloudFormation template could not be resolved: We don't support attributes of the 'AWS::ElastiCache::CacheCluster' resource. This is a CDK limitation. Please report it at https://github.com/aws/aws-cdk/issues/new/choose
Falling back to doing a full deployment
CdkWatchMinimalEcsServiceStack: creating CloudFormation changeset...
[··························································] (0/4)

4:59:14 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack                | CdkWatchMinimalEcsServiceStack