[Bug]: Error creating CloudWatch Logs Log Group with KMS key

[AlexanderRasmussen]

Terraform Core Version

1.3.2

AWS Provider Version

4.52.0

Affected Resource(s)

resource "aws_elasticache_replication_group" resource "aws_cloudwatch_log_group" resource "aws_kms_key"

Expected Behavior

Elasticache should be created with encrypted cloudwatch log groups using the created KMS key to encrypt the logs send to cloudwatch.

Actual Behavior

Fails during apply telling me that the KMS key does not exist or does not have proper configuration. The error is pasted below:

Relevant Error/Panic Output Snippet

Error: Provider produced inconsistent final plan

When expanding the plan for
module.redis.aws_elasticache_user_group.redis to include new
values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for
.user_ids: planned set element cty.StringVal("redisuserid") does
not correlate with any element in actual.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Creating CloudWatch Log Group failed: AccessDeniedException: The specified KMS key does not exist or is not allowed to be used with LogGroup 'arn:aws:logs:eu-central-1:RedactedAccountID:log-group:/redis/slow-log'
    status code: 400, request id: RedactedRequestId '/redis/slow-log'

Terraform Configuration Files

resource "aws_kms_key" "redis_kms_key" {  
  description             = "redis logs kms key"
  key_usage               = "ENCRYPT_DECRYPT"
  enable_key_rotation     = true  
  deletion_window_in_days = 7
  policy                  = data.aws_iam_policy_document.redis_kms_policy.json
}

resource "aws_cloudwatch_log_group" "redis_slow_log" {
  name = "/redis/slow-log"
  kms_key_id = aws_kms_key.redis_kms_key.arn
}

resource "aws_cloudwatch_log_group" "redis_engine_log" {
  name = "/redis/engine-log"
  kms_key_id = aws_kms_key.redis_kms_key.arn
}

resource "random_password" "this" {
  length           = 24
  special          = true
  override_special = "!#$%&*()-_=+[]{}<>:?"
}

resource "aws_elasticache_user" "redis" {
  user_id       = "redisuserid"
  user_name     = "default"
  access_string = "on ~app::* -@all +@read +@hash +@bitmap +@geo -setbit -bitfield -hset -hsetnx -hmset -hincrby -hincrbyfloat -hdel -bitop -geoadd -georadius -georadiusbymember"
  engine        = "REDIS"
  passwords     = [random_password.this.result]
}

resource "aws_elasticache_user_group" "redis" {
  engine        = "REDIS"
  user_group_id = "redisusergroupid"
  user_ids      = [aws_elasticache_user.redis.user_id]
}

resource "aws_elasticache_replication_group" "redis" {
  preferred_cache_cluster_azs = ["eu-central-1a", "eu-central-1b", "eu-central-1c"]
  replication_group_id        = "redis"
  description                 = "Redis"
  node_type                   = "cache.t3.small"
  num_cache_clusters          = 3
  parameter_group_name        = "default.redis7"
  port                        = 6379
  engine                      = "redis"
  engine_version              = "7.0"
  maintenance_window          = "sun:04:00-sun:05:00"
  snapshot_window             = "01:00-02:00"
  subnet_group_name           = aws_elasticache_subnet_group.redis_subnets.name

  multi_az_enabled            = true
  automatic_failover_enabled  = true

  at_rest_encryption_enabled = true
  kms_key_id = aws_kms_key.redis_kms_key.arn

  transit_encryption_enabled = true
  user_group_ids = [ aws_elasticache_user_group.redis.id ]

  notification_topic_arn = "arn:aws:sns:eu-central-1:${var.account_id}:redis"

  log_delivery_configuration {
    destination      = aws_cloudwatch_log_group.redis_slow_log.name
    destination_type = "cloudwatch-logs"
    log_format       = "text"
    log_type         = "slow-log"
  }

  log_delivery_configuration {
    destination      = aws_cloudwatch_log_group.redis_engine_log.name
    destination_type = "cloudwatch-logs"
    log_format       = "text"
    log_type         = "engine-log"
  }
}

resource "aws_elasticache_subnet_group" "redis_subnets" {
  name       = "redis-subnet"
  subnet_ids = var.subnets
}

data "aws_iam_policy_document" "redis_kms_policy" {
  statement {
    sid       = "Enable IAM User Permissions"
    effect    = "Allow"
    actions   = ["kms:*"]
    resources = ["*"]

    principals {
      identifiers = ["arn:aws:iam::${var.account_id}:root"]
      type        = "AWS"
    }
  }

  statement {
    sid    = "Allow access for Key Administrators"
    effect = "Allow"

    actions = [
      "kms:Create*",
      "kms:Describe*",
      "kms:Enable*",
      "kms:List*",
      "kms:Put*",
      "kms:Update*",
      "kms:Revoke*",
      "kms:Disable*",
      "kms:Get*",
      "kms:Delete*",
      "kms:TagResource",
      "kms:UntagResource",
      "kms:ScheduleKeyDeletion",
      "kms:CancelKeyDeletion",
    ]

    resources = ["*"]

    principals {
      identifiers = ["arn:aws:iam::${var.account_id}:role/infra/Configurator"]
      type        = "AWS"
    }
  }

  statement {
    sid    = "Allow logs to encrypt/decrypt messages"
    effect = "Allow"

    actions = [
        "kms:Encrypt*",
        "kms:Decrypt*",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:Describe*"
    ]
    resources = [ "*" ]

    principals {
      identifiers = [ "logs.${var.default_region}.amazonaws.com" ]
      type = "Service"
    }

    condition {
      test = "ArnEquals"
      variable = "kms:EncryptionContext:aws:logs:arn"
      values = ["arn:aws:logs:${var.default_region}:${var.account_id}:log-group:/redis/slow-log", "arn:aws:logs:${var.default_region}:${var.account_id}:log-group:/redis/engine-log"]
    }

    condition {
      test = "StringEquals"
      variable = "kms:ViaService"
      values = ["logs.${var.default_region}.amazonaws.com"]
    }
  }

  statement {
    sid    = "Allow use of key for infrastructure roles within organization"
    effect = "Allow"

    actions = [
      "kms:Encrypt",
      "kms:ReEncrypt*",
      "kms:GenerateDataKey*",
      "kms:DescribeKey",
      "kms:CreateGrant",
      "kms:Decrypt",
    ]

    resources = ["*"]

    principals {
      identifiers = ["*"]
      type        = "AWS"
    }

    condition {
      test     = "StringEquals"
      variable = "kms:ViaService"
      values   = ["elasticache.${var.default_region}.amazonaws.com"]
    }

    condition {
      test     = "StringLike"
      variable = "aws:PrincipalArn"

      values = [
        "arn:aws:iam::*:role/infra/SecurityManager",
        "arn:aws:iam::*:role/infra/CloudManager",
        "arn:aws:iam::*:role/infra/Configurator",
        "arn:aws:iam::*:role/AtlantisConfigurator"
      ]
    }
  }
}

Steps to Reproduce

Take the terraform files pasted above and add region, subnets and account id to it as variable and try to apply it.

Debug Output

No response

Panic Output

No response

Important Factoids

It is very important to know that if i remove the following lines from the KMS policy

    condition {
      test = "ArnEquals"
      variable = "kms:EncryptionContext:aws:logs:arn"
      values = ["arn:aws:logs:${var.default_region}:${var.account_id}:log-group:/redis/slow-log", "arn:aws:logs:${var.default_region}:${var.account_id}:log-group:/redis/engine-log"]
    }

    condition {
      test = "StringEquals"
      variable = "kms:ViaService"
      values = ["logs.${var.default_region}.amazonaws.com"]
    }

Then apply the terraform, readd these lines, and apply it again, there is no problems at all.

References

This guy might have experienced similar issues: https://stackoverflow.com/questions/74014645/how-to-add-the-kmscalleraccount-condition-to-a-kms-key-used-for-encrypting-cl

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[avazula]

Bump this issue is still valid as of Feb 2024. I see that it's been triaged, is there a fix planned somewhere in the roadmap?

[GNSunny]

Terraform Version : 1.7.0

AWS Provider Version : 5.21.0

Affected Resource(s)

resource "aws_cloudwatch_log_group" resource "aws_kms_key"

I can still see this error at the moment.

Error: creating CloudWatch Logs Log Group (vpc-flow-logs): AccessDeniedException: The specified KMS key does not exist or is not allowed to be used with Arn 'arn:aws:logs:eu-west-2:1234567890:log-group:vpc-flow-logs'

Even though it has enough permissions, it is still not working.

[filpacnap]

I had the same error try adding this policy to the kms key :)

resource "aws_kms_key" "apigw_kms" {
  description             = "This key is for cloudwatch logs in apigw"
  deletion_window_in_days = 10
  enable_key_rotation     = true

  policy = jsonencode({
    "Version": "2012-10-17",
    "Id": "default",
    "Statement": [
      {
        "Sid": "DefaultAllow",
        "Effect": "Allow",
        "Principal": {
          "AWS": "arn:aws:iam::${var.account_id}:root"
        },
        "Action": "kms:*",
        "Resource": "*"

      },
      {
        Effect    = "Allow"
        Principal = {
          Service = "logs.${var.region}.amazonaws.com"
        }
        Action    = [
          "kms:*"
        ]
        Resource  = "*"
      }
    ]
  })
}

[ericluria]

@filpacnap is correct, this is an IAM issue and not a Terraform issue!