Kinesis Analytics application updating runtime environment throws error "ConcurrentModificationException: Tags are already registered"

[spachari-im]

Terraform Core Version

1.1.7

AWS Provider Version

4.66.1

Affected Resource(s)

• aws_kinesis_analytics_application

Expected Behavior

After the issue https://github.com/hashicorp/terraform-provider-aws/issues/14541 was fixed in AWS provider version 3.14.0, when the runtime_environment attribute of the aws_kinesis_analytics_application resource is changed from FLINK-1_13 to FLINK-1_15, Terraform should delete the existing AWS Kinesis Analytics application and create a new application with the new runtime environment.

Actual Behavior

When Terraform apply is run, the AWS Kinesis application is deleted, and apply step fails with the following error thrown.

running "/usr/local/bin/terraform apply -input=false \"/atlantis-data/repos/tn/terraform-d/169/media-dev/terraform/main-infra/bd/Bid_Pacer_Dev-media-dev.tfplan\"" in "/atlantis-data/repos/tn/terraform-d/169/media-dev/terraform/main-infra/bd": exit status 1
module.bd.aws_kinesisanalyticsv2_application.app: Destroying... [id=arn:aws:kinesisanalytics:us-east-1:<aws_account_id>:application/d-bd-Dev]
module.bd.aws_kinesisanalyticsv2_application.app: Still destroying... [id=arn:aws:kinesisanalytics:us-east-1:<aws_account_id>:application/d-bd-Dev, 10s elapsed]
module.bd.aws_kinesisanalyticsv2_application.app: Still destroying... [id=arn:aws:kinesisanalytics:us-east-1:<aws_account_id>:application/d-bd-Dev, 20s elapsed]
module.bd.aws_kinesisanalyticsv2_application.app: Still destroying... [id=arn:aws:kinesisanalytics:us-east-1:<aws_account_id>:application/d-bd-Dev, 30s elapsed]
module.bd.aws_kinesisanalyticsv2_application.app: Still destroying... [id=arn:aws:kinesisanalytics:us-east-1:<aws_account_id>:application/d-bd-Dev, 40s elapsed]
module.bd.aws_kinesisanalyticsv2_application.app: Destruction complete after 43s
module.bd.aws_kinesisanalyticsv2_application.app: Creating...
╷
│ Error: creating Kinesis Analytics v2 Application (d-bd-Dev): ConcurrentModificationException: Tags are already registered for this resource ARN: arn:aws:kinesisanalytics:us-east-1:<aws_account_id>:application/d-bd-Dev, please retry later. Or you can create without tags and then add tags using TagResource API after successful resource creation.
│ 
│   with module.bd.aws_kinesisanalyticsv2_application.app,
│   on ../../modules/bd/main.tf line 57, in resource "aws_kinesisanalyticsv2_application" "app":
│   57: resource "aws_kinesisanalyticsv2_application" "app" {
│ 
╵

If I run the apply step again after 15-20 minutes of the failure in the first apply step, the new application is created.

Relevant Error/Panic Output Snippet

 Error: creating Kinesis Analytics v2 Application (d-bd-Dev): ConcurrentModificationException: Tags are already registered for this resource ARN: arn:aws:kinesisanalytics:us-east-1:<aws_account_id>:application/d-bd-Dev, please retry later. Or you can create without tags and then add tags using TagResource API after successful resource creation.

   with module.bd.aws_kinesisanalyticsv2_application.app,
   on ../../modules/bd/main.tf line 57, in resource "aws_kinesisanalyticsv2_application" "app":
   57: resource "aws_kinesisanalyticsv2_application" "app" {

Terraform Configuration Files

provider "aws" {
  region  = "us-east-1"
  profile = media-dev
  default_tags {
    tags = {
      Region  = "us-east-1"
    }
  }
}

resource "aws_kinesisanalyticsv2_application" "app" {
  name                   = "d-bd-Dev"
  runtime_environment    = "FLINK-1_15"
  service_execution_role = aws_iam_role.kinesis_application_role.arn
  start_application      = false
  force_stop             = true

  application_configuration {
    application_code_configuration {
      code_content {
        s3_content_location {
          bucket_arn     = local.kinesis_application_jar_details.BucketARNUpdate
          file_key       = local.kinesis_application_jar_details.FileKeyUpdate
          object_version = local.kinesis_application_jar_details.ObjectVersionUpdate
        }
      }

      code_content_type = "ZIPFILE"
    }

    flink_application_configuration {
      checkpoint_configuration {
        configuration_type            = "CUSTOM"
        checkpointing_enabled         = true
        checkpoint_interval           = 300000
        min_pause_between_checkpoints = 60000
      }
    }
  }
}

Steps to Reproduce

• Install an AWS Kinesis Analytics with runtime_environment = "FLINK-1_13" through Terraform
• Modify the Terraform file to change the runtime_environment attribute to "FLINK-1_15"
• Run plan and apply again

Debug Output

No response

Panic Output

No response

Important Factoids

This issue is described in https://github.com/hashicorp/terraform-provider-aws/issues/14541 and https://github.com/hashicorp/terraform-provider-aws/issues/15881. A fix was released in version 3.14.0 of the Terraform AWS provider.

References

https://github.com/hashicorp/terraform-provider-aws/issues/14541 https://github.com/hashicorp/terraform-provider-aws/issues/15881

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[josacar]

I can confirm create -> delete -> recreate same flink app with same parameters just right after destruction ends into this issue.