Potentially cached values in cdk.context.json can break deployments

[whummer]

Seems like CDK is caching values in cdk.context.json that are not always updated on cdk bootstrap or cdk deploy.

For example, just ran into a case where cdk.context.json contained cached local VPC/subnet details:

{
  "vpc-provider:account=000000000000:filter.isDefault=true:region=us-east-1:returnAsymmetricSubnets=true": {
    "vpcId": "vpc-69bd3613",
    "vpcCidrBlock": "172.31.0.0/16",
    "availabilityZones": [],
    "subnetGroups": [
      {
        "name": "Public",
        "type": "Public",
        "subnets": [
          {
            "subnetId": "subnet-47b445b3",
            "cidr": "172.31.0.0/20",
            "availabilityZone": "us-east-1a",
            "routeTableId": "rtb-b807fb5a"
          },
          {
            "subnetId": "subnet-99289971",
            "cidr": "172.31.16.0/20",
            "availabilityZone": "us-east-1b",
            "routeTableId": "rtb-b807fb5a"
          },
  ...

... and upon deployment of an AWS::RDS::DBSubnetGroup resource that was referencing one of the subnets, the deployment failed, as this subnet was no longer returned from awslocal ec2 describe-subnets (double-checked that the region was correct).

botocore.exceptions.ClientError: An error occurred (InvalidSubnetID.NotFound) when calling the CreateDBSubnetGroup operation: The subnet ID 'subnet-47b445b3' does not exist

We should consider deleting the cdk.context.json cache file (or updating its entries) on cdklocal bootstrap and/or cdklocal deploy, to ensure we're not using stale entries for deployment..

/cc @dominikschubert

[dominikschubert]

:+1: Definitely a good idea to build some fail-safes for this.

Another cache that has caused issues in the past is ~/.cdk/cache :thinking:

[NinoSkopac]

I fixed tihs issue by manually deleting cdk.context.json

[mwjp]

For anyone else who comes searching, I have been using process.env.CDK_DEFAULT_ACCOUNT within my code without issue (for reference, this environment variable carries the AWS account number of the account CDK is acting against). Today it stopped working, in that it would only resolve to undefined (with no code changes of any kind that would affect this, and shell restarts had no effect). I tried re-authenticating my SSO credentials to no avail.

Making absolutely zero changes of any kind, I just deleted cdk.context.json. When I re-ran, I got SSOTokenProviderFailure: SSO Token refresh failed. Please log in using "aws sso login". The session was still perfectly valid (confirmed with other AWS CLI commands), but I went ahead and obliged and re-authenticated once again. After that, all was well.

This is not directly related to the original content of the subnet thing, but, is just another example of cdk.context.json cached values causing issues, so thought I'd leave this as it's the first Google result for cdk.context.json cache issues.

(Aaaand I just realized this isn't the official aws-cdk repo 😆 Oh well, as this tool seems to use cdk.context.json as well, I'll leave it as it may be relevant anyway).

[lakkeger]

It seems context is only updated during synth to make sure ie your previously "latest" version of AMI not updated and result in a new provision of your EC2 instances. Now this seems a reasonable assumption in a live AWS environment, however by restarting LocalStack we might end up with undesired values in the context like @whummer described it. To clear this, AWS recommends to run cdk context --clear to empty the context cache or for single values cdk context --reset <KEY_OR_NUMBER>.

Now since most users wouldn't need the cache with LocalStack I guess it's reasonable to add the context clear command run as a default behaviour and for those who need it lets provide an env variable to keep it, ie SKIP_CONTEXT_CLEAR=1.

Ref to the relevant docs of CDK: https://docs.aws.amazon.com/cdk/v2/guide/context.html

/cc @dominikschubert