[Bug]: unable to upgrade elastic cache version from terraform

[shettypriy]

Terraform Core Version

1.6.4

AWS Provider Version

5.42.0

Affected Resource(s)

aws_elasticache_replication_group

Expected Behavior

should be able to upgrade the version to 7

Actual Behavior

Failing with following error

Error: parsing old engine_version: Malformed version: 7.x
│
│   with module.redis_test.aws_elasticache_replication_group.redis_with_auth[0],
│   on ../../../../modules/aws/data/elastic_cache/main.tf line 41, in resource "aws_elasticache_replication_group" "redis_with_auth":
│   41: resource "aws_elasticache_replication_group" "redis_with_auth" {

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

module "redis_test" {
  source                         = "../../../../modules/aws/data/elastic_cache"
  env                            = var.env
  name                           = "temp"
  redis_clusters                 = "1"
  redis_failover                 = "false"
  redis_node_type                = "cache.t3.micro"
  subnet_group_name              = local.cache_subnet
  vpc_id                         = local.vpc_id
  redis_maintenance_window       = "sat:04:00-sat:05:00"
  redis_version                  = "7.0"
  redis_snapshot_retention_limit = "1"
  region                         = var.aws_region
  at_rest_encryption_enabled     = true
  transit_encryption_enabled     = true
  auth_enabled                   = true
  apply_immediately              = true
  snapshot_name                  = ""
}

It planned and then gave below error

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with
the following symbols:
-/+ destroy and then create replacement

Terraform planned the following actions, but then encountered a problem:

# module.redis_test.aws_elasticache_parameter_group.redis_parameter_group must be replaced
-/+ resource "aws_elasticache_parameter_group" "redis_parameter_group" {
      ~ arn         = "****************************" -> (known after apply)
      ~ family      = "redis6.x" -> "redis7" # forces replacement
      ~ id          = "temp" -> (known after apply)
        name        = "temp"

        # (2 unchanged attributes hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Error: parsing old engine_version: Malformed version: 7.x
│
│   with module.redis_test.aws_elasticache_replication_group.redis_with_auth[0],
│   on ../../../../modules/aws/data/elastic_cache/main.tf line 41, in resource "aws_elasticache_replication_group" "redis_with_auth":
│   41: resource "aws_elasticache_replication_group" "redis_with_auth" {
│

replication group resource:

resource "aws_elasticache_replication_group" "redis_with_auth" {

  replication_group_id       = format("%.20s", "${var.env}-${var.name}")
  description                = "Terraform-managed ElastiCache replication group for ${var.name}"
  num_cache_clusters         = var.redis_clusters
  node_type                  = var.redis_node_type
  automatic_failover_enabled = var.redis_failover
  multi_az_enabled           = var.redis_failover
  engine_version             = var.redis_version
  port                       = var.redis_port
  parameter_group_name       = aws_elasticache_parameter_group.redis_parameter_group.id
  subnet_group_name          = var.subnet_group_name
  security_group_ids         = [aws_security_group.redis_security_group.id]
  apply_immediately          = var.apply_immediately
  maintenance_window         = var.redis_maintenance_window
  snapshot_window            = var.redis_snapshot_window
  snapshot_retention_limit   = var.redis_snapshot_retention_limit
  snapshot_name              = var.snapshot_name
  at_rest_encryption_enabled = var.at_rest_encryption_enabled
  transit_encryption_enabled = var.transit_encryption_enabled
  auth_token                 = var.auth_token

}

Steps to Reproduce

Upgraded elasticcache engine versionfrom 6.2.6 to 7.0 from the AWS console, Now trying to update the engine version in the terraform and getting below error

Error: parsing old engine_version: Malformed version: 7.x

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[acwwat]

Relates #33867

[justinretzolk]

Hey @shettypriy 👋 Thank you for taking the time to raise this! Are you able to supply debug logs (redacted as needed) in case whoever looks into this needs that information?

[amitabh90]

hi @justinretzolk I am also facing exact same issue , Kindly let me know how to gather the debug log and share with you

[darkhelmet]

I'm hitting this as well, here's my lockfile https://gist.github.com/darkhelmet/7689bd5928d5832b75b4970a2aef6ee9

I've already updated the Redis instance to 7.1 via the console, and this is now just running plan after I've updated my TF

[GregorKocevar]

I encountered the same issue while updating my Terraform AWS provider. My Redis cache instances were originally created using AWS provider version 3, and the engine version I used was specified as "7.x". When I attempted to update the provider to the latest version (5.54.1), I encountered the error related to parsing the engine_version.

To resolve this issue, I initially fixed it by manually modifying the tfstate file. However, this approach is not scalable for my projects because I have a large number of Redis cache instances, making it impractical to edit each tfstate file individually.

[p2isecke]

We were able to reproduce this issue and believe it's caused by this regex check when matches on [6-9]\.x when it should only match on 6\.x.

Reproduction steps

1. Create an aws_elasticache_replication_group resource to use engine_version = "6.x"

   resource "aws_elasticache_replication_group" "redis" {
   replication_group_id = "redis"
   description = "redis-cluster"
   subnet_group_name = aws_elasticache_subnet_group.redis.name
   security_group_ids   = [aws_security_group.redis-instance-sg.id]
   node_type            = "cache.t2.small"
   transit_encryption_enabled = true
   apply_immediately = true
   
   engine_version = "6.x"
   }

2. Manually upgrade the cluster in the console to 7.1
3. Update the terraform code so that engine_version = "7.1"
4. Error thrown: Error: parsing old engine_version: Malformed version: 7.x

Investigation

setEngineVersionRedis is responsible for setting the engine_version attribute. This function gets called by resourceGlobalReplicationGroup() which performs an AWS API query via resourceGlobalReplicationGroupRead(). By looking up the Global Replication Group, the EngineVersion that gets passed to setEngineVersionRedisis 7.1 However, configVersion in this function is actually 6.x because d.Get(names.AttrEngineVersion).(string) retrieved from the state file is 6.x. Because this regex evaluation is true, we update the engine_version attribute using the value passed into setEngineVersionRedis, which is 7.1. As a result, we get a version of 7.x

We believe this was introduced as part of provider v5.3.0 release. Prior to 5.3.0, the plan would have been successful and proposed ~ engine_version = "7.x" -> "7.1" # forces replacement

Proposal

Update the regex evaluation from if t, _ := regexp.MatchString([6-9].x, configVersion); t { to if t, _ := regexp.MatchString(6.x, configVersion); t {.

We tested this locally and this seemed to have prevented the malformed error and the plan succeeded.