AWS::ElastiCache::ReplicationGroup - AWS::ElastiCache - Unable to scale up a cluster

[ueokande]

1. Title

AWS::ElastiCache - Unable to scale up a cluster

2. Scope of request

e) other coverage-related issue with the resource/attribute/option

3. Expected behavior

I have a Redis cluster with a single node. I tried to scale up the cluster to a multi-node cluster by CloudFormation stack. But it fails on applying the stack.

Current cluster have single node without automatic failover. The cluster is created by the following CloudFormation resource:

MyRedisCludster:
  Type: "AWS::ElastiCache::ReplicationGroup"
  Properties:
    ReplicationGroupDescription: "My cluster"
    EngineVersion: "5.0.5"
    CacheNodeType: "cache.t3.micro"
    Engine: "redis"
    NumCacheClusters: 1
    AutomaticFailoverEnabled: "false"

Then I tried to scale-up the cluster: increase to two of nodes, and enable automatic failover by the following:

MyRedisCludster:
  Type: "AWS::ElastiCache::ReplicationGroup"
  Properties:
    ReplicationGroupDescription: "My cluster"
    EngineVersion: "5.0.5"
    CacheNodeType: "cache.t3.micro"
    Engine: "redis"
    NumCacheClusters: 2  # <-- Update
    AutomaticFailoverEnabled: "true"  # <-- Update

The stack will fail and says the following message:

Replication group must have at least one read replica to enable autofailover.

The stack should complete updating the cluster successfully.

4. Suggest specific test cases

1. Create a cluster with a single-node.
2. Modify the template to scale-up to two-nodes cluster and failover enabled.
3. Apply the template.
4. The stack finishes with no errors
5. The cluster has two nodes and the failover is enabled.

5. Helpful Links to speed up research and evaluation

6. Category (required) - Will help with tagging and be easier to find by other users to +1

3. DB (ElastiCache)

7. Any additional context (optional)

[luiseduardocolon]

Just for kicks, have you tried the first change (increasing the cluster size) in one update operation, and then try the 2nd change (enable auto failover) once the state recognizes that there is more than one cluster?

[ueokande]

@luiseduardocolon Yes, I have applied two changes, increasing the cluster and enabling failover at once. When I try to apply them with two operations (increasing nodes, then enable failover), they had completed successfully.

I guess the stacks should apply them at a time if cloudformation abides by the principles of declarative infrastructure.

[luiseduardocolon]

Thanks for trying that. Just wondering if the behavior is the same at the API level. If you can replicate the issue (getting the same error message via CLI/API), then the issue is with the downstream API not allowing both changes. If the downstream API does allow both changes in one call, then the issue might be with cloudformation.

[ueokande]

Although CloudFormation allows changing NumCacheClusters and AutomaticFailoverEnabled in a single resource, actually the APIs to modify the cluster are separated two. Increasing the cluster (i.e. adding a node) is applied via CreateCacheCluster API and enabling automatic failover is done via ModifyReplicationGroup.

To modify the cluster by aws-cli is the following:

1. Add a node to the cluster:

   aws elasticache create-cache-cluster --cache-cluster-id my-cluster-001 \
   --replication-group-id my-cluster

2.Enable automatic failover:

aws elasticache modify-replication-group --replication-group-id my-cluster \
  --automatic-failover-enabled

[ipmb]

I'm seeing the same thing with MultiAZEnabled. Going from 1 to 2 on NumCacheClusters and setting MultiAZEnabled in the same update fails.