[Bug]: IAM Role with policies created but permissions not available for dependent resources

[RubenMakandra]

Terraform Core Version

1.5.3

AWS Provider Version

4.55.0

Affected Resource(s)

• aws_sesv2_configuration_set_event_destination

Expected Behavior

• IAM Role is created
• IAM Policy is created
• IAM Policy is attached
• aws_sesv2_configuration_set_event_destination is created
• aws_sesv2_configuration_set_event_destination is created

Actual Behavior

• IAM Role is created
• IAM Policy is created
• IAM Policy is attached
• aws_sesv2_configuration_set_event_destination is created
• aws_sesv2_configuration_set_event_destination is not created, because the IAM permissions are not available yet

Relevant Error/Panic Output Snippet

│ Error: creating Amazon SESv2 (Simple Email V2) Configuration Set Event Destination (test_Set|kinesis_event_destination): operation error SESv2: CreateConfigurationSetEventDestination, https response error StatusCode: 400, RequestID: b2430249-db6b-473f-bebc-d4f33dac4a3e, BadRequestException: Could not access Kinesis Firehose Stream <arn:aws:firehose:REGION_REDACTED:ACCOUNT_REDACTED::deliverystream/STREAM_NAME> using IAM role <arn:aws:iam::ACCOUNT_REDACTED:role/SES_access_to_Kinesis>
│
│   with aws_sesv2_configuration_set_event_destination.bounce_log_kinesis_stream,
│   on firehose.tf line 106, in resource "aws_sesv2_configuration_set_event_destination" "bounce_log_kinesis_stream":
│  106: resource "aws_sesv2_configuration_set_event_destination" "bounce_log_kinesis_stream" {
│
╵

Terraform Configuration Files

locals {
  kinesis_firehose_stream_arn      = "STREAM_ARN"
}

data "aws_iam_policy_document" "ses_assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["ses.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}

data "aws_iam_policy_document" "ses_access_to_kinesis" {
  statement {
    effect = "Allow"
    actions = [
      "firehose:PutRecord",
      "firehose:PutRecordBatch",
    ]
    resources = [
      local.kinesis_firehose_stream_arn
    ]
  }
}

resource "aws_iam_policy" "ses_access_to_kinesis" {
  name   = "SES_access_to_Kinesis"
  policy = data.aws_iam_policy_document.ses_access_to_kinesis.json
}

resource "aws_iam_role" "ses_access_to_kinesis" {
  name               = "SES_access_to_Kinesis"
  assume_role_policy = data.aws_iam_policy_document.ses_assume_role.json
}

resource "aws_iam_role_policy_attachment" "ses_access_to_kinesis" {
  policy_arn = aws_iam_policy.ses_access_to_kinesis.arn
  role       = aws_iam_role.ses_access_to_kinesis.name

}

resource "aws_sesv2_configuration_set" "bounce_log" {
  configuration_set_name = "test_Set"
  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_sesv2_configuration_set_event_destination" "bounce_log_kinesis_stream" {
  configuration_set_name = aws_sesv2_configuration_set.bounce_log.configuration_set_name
  event_destination_name = "kinesis_event_destination"

  event_destination {
    kinesis_firehose_destination {
      delivery_stream_arn = local.kinesis_firehose_stream_arn
      iam_role_arn        = aws_iam_role.ses_access_to_kinesis.arn
    }

    enabled              = true
    matching_event_types = ["BOUNCE"]
  }
}

Steps to Reproduce

1. Create/have an AWS kinesis delivery stream and obtain it's arn
2. Add the stream ARN as local to the given configuration file (line 2)
3. run terraform apply

Debug Output

No response

Panic Output

No response

Important Factoids

Running terraform applya second time successfully creates the aws_sesv2_configuration_set_event_destination. I assume the underlying reason for this behaviour is the same as in the referenced ticked #14008, and the solution would be to retry the resource creation, similar to the fix implemented for that issue.

References

14008

Would you like to implement a fix?

No

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[dpowley]

Can you use the depends_on meta argument?

Something like:

resource "aws_sesv2_configuration_set_event_destination" "bounce_log_kinesis_stream" {
  configuration_set_name = aws_sesv2_configuration_set.bounce_log.configuration_set_name
  event_destination_name = "kinesis_event_destination"

  event_destination {
    kinesis_firehose_destination {
      delivery_stream_arn = local.kinesis_firehose_stream_arn
      iam_role_arn        = aws_iam_role.ses_access_to_kinesis.arn
    }

    enabled              = true
    matching_event_types = ["BOUNCE"]
  }
  depends_on = [aws_iam_role_policy_attachment. ses_access_to_kinesis]
}

[grahamhar]

Can you use the depends_on meta argument?

Something like:

resource "aws_sesv2_configuration_set_event_destination" "bounce_log_kinesis_stream" {

  configuration_set_name = aws_sesv2_configuration_set.bounce_log.configuration_set_name

  event_destination_name = "kinesis_event_destination"

  event_destination {

    kinesis_firehose_destination {

      delivery_stream_arn = local.kinesis_firehose_stream_arn

      iam_role_arn        = aws_iam_role.ses_access_to_kinesis.arn

    }

    enabled              = true

    matching_event_types = ["BOUNCE"]

  }

  depends_on = [aws_iam_role_policy_attachment. ses_access_to_kinesis]

}

The other option would be to use an managed_policy_arns on the role directly rather than attaching them separately , as the dependency on the role ARN is already there this should also solve the issue.

[RubenMakandra]

Can you use the depends_on meta argument?

Something like:

resource "aws_sesv2_configuration_set_event_destination" "bounce_log_kinesis_stream" {
  configuration_set_name = aws_sesv2_configuration_set.bounce_log.configuration_set_name
  event_destination_name = "kinesis_event_destination"

  event_destination {
    kinesis_firehose_destination {
      delivery_stream_arn = local.kinesis_firehose_stream_arn
      iam_role_arn        = aws_iam_role.ses_access_to_kinesis.arn
    }

    enabled              = true
    matching_event_types = ["BOUNCE"]
  }
  depends_on = [aws_iam_role_policy_attachment. ses_access_to_kinesis]
}

Thanks for the suggestion, I tried it out, and get a different IAM related error, in 3/3 tries:

│ Error: creating Amazon SESv2 (Simple Email V2) Configuration Set Event Destination (test_Set|kinesis_event_destination): operation error SESv2: CreateConfigurationSetEventDestination, https response error StatusCode: 400, RequestID: f230abda-d9ba-4b30-9121-d71639027a02, BadRequestException: Could not assume IAM role <arn:aws:iam::ARN:role/SES_access_to_Kinesis>.
│ 
│   with aws_sesv2_configuration_set_event_destination.bounce_log_kinesis_stream,
│   on test.tf line 54, in resource "aws_sesv2_configuration_set_event_destination" "bounce_log_kinesis_stream":
│   54: resource "aws_sesv2_configuration_set_event_destination" "bounce_log_kinesis_stream" {
│ 

Applying a second time worked without issue.
I also tried @grahamhar suggestion and unsurprisingly received the same error message, that the role couldn't be assumed.
I can also post an updated terraform code sample that also creates a bucket and the firehose delivery stream, if it would be helpful. Issue there is that creating the bucket and stream makes the terraform apply a lot slower, that's why I just used a reference to an existing stream in the example and in my tests.

[tophercullen]

Running into the same exact issue/case. My first thought was also to add an explicate depends_on for the event destination resource . That didn't have any affect.

Looking over the high level resource log output, the ordering is indeed correct. It waits until the policy attachment completes before attempting to create the aws_sesv2_configuration_set_event_destination. It fails just the same. Re-running and it works fine.

Looking through the cloudtrail logs, it seems like there's a similar condition with the Kinesis stream. The cloutrail logs show the create fails several times, being unable to assume its prescribed IAM role, and then it magically works. Looking over the AWS provider code for Kinesis, it seems like that resource has built-in retry logic to handles this delay in dependent resources being available: https://github.com/hashicorp/terraform-provider-aws/blob/9e149726e119b2c8cde94b2f3f5de225b1344b50/internal/service/firehose/delivery_stream.go#L1506

[plain5]

• I faced the same issue as described here. time_sleep helped to solve the problem. Example of the configuration :

resource "time_sleep" "wait_120_seconds" {
  depends_on = [aws_iam_role_policy_attachment.ses_access_to_kinesis]

  create_duration = "120s"
}

resource "aws_sesv2_configuration_set_event_destination" "bounce_log_kinesis_stream" {
  configuration_set_name = aws_sesv2_configuration_set.bounce_log.configuration_set_name
  event_destination_name = "kinesis_event_destination"

  event_destination {
    kinesis_firehose_destination {
      delivery_stream_arn = local.kinesis_firehose_stream_arn
      iam_role_arn        = aws_iam_role.ses_access_to_kinesis.arn
    }

    enabled              = true
    matching_event_types = ["BOUNCE"]
  }

  depends_on = [time_sleep.wait_120_seconds]
}

• I've found a useful comment about this issue. You can pay attention to it.