[Bug]: "Destination stream validation failed" pinpoint -> kinesis firehose

[loganpowell]

Terraform Core Version

1.3.0

AWS Provider Version

4.32.0

Affected Resource(s)

• aws_kinesis_firehose_delivery_stream
• aws_pinpoint_event_stream

Expected Behavior

I found an issue reporting the same error, which was closed, but I believe I may be experiencing it still...

I'm trying to configure Pinpoint to send events to a Kinesis Firehose and have tried a number of variations of roles/policies and settings - all of which fail.

Actual Behavior

Terraform performs beautifully on everything else then chokes on the Pinpoint Stream put op.

Relevant Error/Panic Output Snippet

...
aws_pinpoint_event_stream.pinpoint_firehose: Creating...
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [10s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [20s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [30s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [40s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [50s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [1m0s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [1m10s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [1m20s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [1m30s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [1m40s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [1m50s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [2m0s elapsed]
╷
│ Error: error putting Pinpoint Event Stream for application xxxxxxxxxxxxxxx: BadRequestException: Destination stream validation failed. Check Destination and Role ARN values and make sure the IAM Role is configured correctly.
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "xxxxxxxxxxxxxxxxx"
│   },
│   Message_: "Destination stream validation failed. Check Destination and Role ARN values and make sure the IAM Role is configured correctly.",
│   RequestID_: "xxxxxxxxxxxxxxxxx"
│ }
│
│   with aws_pinpoint_event_stream.pinpoint_firehose,
│   on main.tf line 188, in resource "aws_pinpoint_event_stream" "pinpoint_firehose":
│  188: resource "aws_pinpoint_event_stream" "pinpoint_firehose" {

Terraform Configuration Files

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.32.0"
    }
  }
}

provider "aws" {
  alias   = "email"
  region  = "us-east-1"
  profile = "default"
}

resource "aws_iam_role" "firehose_role" {
  name = "iam_firehose_role"
  assume_role_policy = jsonencode({
    Version : "2012-10-17",
    Statement : [
      {
        Effect : "Allow",
        Action : "sts:AssumeRole",
        Principal : {
          Service : "firehose.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_role_policy" "firehose_policy" {
  name = "pinpoint_firehose_policy"
  role = aws_iam_role.firehose_role.id
  policy = jsonencode({
    Version : "2012-10-17",
    Statement : [
      {
        Effect : "Allow",
        Action : [
          "firehose:PutRecordBatch",
          "firehose:DescribeDeliveryStream"
        ],
        Resource : [
          "arn:aws:firehose:us-east-1:<account id>:deliverystream/*"
        ]
      }
    ]
  })
}

resource "aws_s3_bucket" "firehose_events" {
  bucket = "pinpoint-error"
}
resource "aws_s3_bucket_acl" "bucket_acl" {
  bucket = aws_s3_bucket.firehose_events.id
  acl    = "private"
}

resource "aws_kinesis_firehose_delivery_stream" "fh_stream" {
  name        = "firehose"
  destination = "extended_s3"
  extended_s3_configuration {
    role_arn            = aws_iam_role.firehose_role.arn
    bucket_arn          = aws_s3_bucket.firehose_events.arn
    prefix              = "!{timestamp:yyyy/MM/dd}"
    error_output_prefix = "!{timestamp:yyyy/MM/dd}/!{firehose:error-output-type}"
    buffer_interval     = 360 # range: 60 to 900 seconds
    buffer_size         = 64  # range: 64 to 128 MiB
  }
}

resource "aws_pinpoint_app" "my_app" {
  name = "elvis_lives"
}
resource "aws_pinpoint_event_stream" "pinpoint_firehose" {
  application_id         = aws_pinpoint_app.my_app.id
  destination_stream_arn = aws_kinesis_firehose_delivery_stream.fh_stream.arn
  role_arn               = aws_iam_role.firehose_role.arn
  depends_on = [
    aws_iam_role.firehose_role,
    aws_iam_role_policy.firehose_policy,
    aws_kinesis_firehose_delivery_stream.fh_stream
  ]
}

Steps to Reproduce

replace <account id> in the aws_iam_role_policy.firehose_policy with any account id and run terraform apply

Debug Output

λ terraform apply
aws_iam_role.firehose_role: Refreshing state... [id=iam_firehose_role]
aws_pinpoint_app.my_app: Refreshing state... [id=xxxxxxxxxx]
aws_iam_role_policy.firehose_policy: Refreshing state... [id=iam_firehose_role:pinpoint_firehose_policy]

Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_kinesis_firehose_delivery_stream.fh_stream will be created
  + resource "aws_kinesis_firehose_delivery_stream" "fh_stream" {
      + arn            = (known after apply)
      + destination    = "extended_s3"
      + destination_id = (known after apply)
      + id             = (known after apply)
      + name           = "firehose"
      + tags_all       = (known after apply)
      + version_id     = (known after apply)

      + extended_s3_configuration {
          + bucket_arn          = (known after apply)
          + buffer_interval     = 360
          + buffer_size         = 64
          + compression_format  = "UNCOMPRESSED"
          + error_output_prefix = "!{timestamp:yyyy/MM/dd}/!{firehose:error-output-type}"
          + prefix              = "!{timestamp:yyyy/MM/dd}"
          + role_arn            = "arn:aws:iam::xxxxxxxx:role/iam_firehose_role"
          + s3_backup_mode      = "Disabled"

          + cloudwatch_logging_options {
              + enabled         = (known after apply)
              + log_group_name  = (known after apply)
              + log_stream_name = (known after apply)
            }
        }
    }

  # aws_pinpoint_event_stream.pinpoint_firehose will be created
  + resource "aws_pinpoint_event_stream" "pinpoint_firehose" {
      + application_id         = "xxxxxxxxxxxxxxxxxxxxxxxxx"
      + destination_stream_arn = (known after apply)
      + id                     = (known after apply)
      + role_arn               = "arn:aws:iam::xxxxxxxx:role/iam_firehose_role"
    }

  # aws_s3_bucket.firehose_events will be created
  + resource "aws_s3_bucket" "firehose_events" {
      + acceleration_status         = (known after apply)
      + acl                         = (known after apply)
      + arn                         = (known after apply)
      + bucket                      = "pinpoint-error"
      + bucket_domain_name          = (known after apply)
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = false
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + object_lock_enabled         = (known after apply)
      + policy                      = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + tags_all                    = (known after apply)
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)

      + cors_rule {
          + allowed_headers = (known after apply)
          + allowed_methods = (known after apply)
          + allowed_origins = (known after apply)
          + expose_headers  = (known after apply)
          + max_age_seconds = (known after apply)
        }

      + grant {
          + id          = (known after apply)
          + permissions = (known after apply)
          + type        = (known after apply)
          + uri         = (known after apply)
        }

      + lifecycle_rule {
          + abort_incomplete_multipart_upload_days = (known after apply)
          + enabled                                = (known after apply)
          + id                                     = (known after apply)
          + prefix                                 = (known after apply)
          + tags                                   = (known after apply)

          + expiration {
              + date                         = (known after apply)
              + days                         = (known after apply)
              + expired_object_delete_marker = (known after apply)
            }

          + noncurrent_version_expiration {
              + days = (known after apply)
            }

          + noncurrent_version_transition {
              + days          = (known after apply)
              + storage_class = (known after apply)
            }

          + transition {
              + date          = (known after apply)
              + days          = (known after apply)
              + storage_class = (known after apply)
            }
        }

      + logging {
          + target_bucket = (known after apply)
          + target_prefix = (known after apply)
        }

      + object_lock_configuration {
          + object_lock_enabled = (known after apply)

          + rule {
              + default_retention {
                  + days  = (known after apply)
                  + mode  = (known after apply)
                  + years = (known after apply)
                }
            }
        }

      + replication_configuration {
          + role = (known after apply)

          + rules {
              + delete_marker_replication_status = (known after apply)
              + id                               = (known after apply)
              + prefix                           = (known after apply)
              + priority                         = (known after apply)
              + status                           = (known after apply)

              + destination {
                  + account_id         = (known after apply)
                  + bucket             = (known after apply)
                  + replica_kms_key_id = (known after apply)
                  + storage_class      = (known after apply)

                  + access_control_translation {
                      + owner = (known after apply)
                    }

                  + metrics {
                      + minutes = (known after apply)
                      + status  = (known after apply)
                    }

                  + replication_time {
                      + minutes = (known after apply)
                      + status  = (known after apply)
                    }
                }

              + filter {
                  + prefix = (known after apply)
                  + tags   = (known after apply)
                }

              + source_selection_criteria {
                  + sse_kms_encrypted_objects {
                      + enabled = (known after apply)
                    }
                }
            }
        }

      + server_side_encryption_configuration {
          + rule {
              + bucket_key_enabled = (known after apply)

              + apply_server_side_encryption_by_default {
                  + kms_master_key_id = (known after apply)
                  + sse_algorithm     = (known after apply)
                }
            }
        }

      + versioning {
          + enabled    = (known after apply)
          + mfa_delete = (known after apply)
        }

      + website {
          + error_document           = (known after apply)
          + index_document           = (known after apply)
          + redirect_all_requests_to = (known after apply)
          + routing_rules            = (known after apply)
        }
    }

  # aws_s3_bucket_acl.bucket_acl will be created
  + resource "aws_s3_bucket_acl" "bucket_acl" {
      + acl    = "private"
      + bucket = (known after apply)
      + id     = (known after apply)

      + access_control_policy {
          + grant {
              + permission = (known after apply)

              + grantee {
                  + display_name  = (known after apply)
                  + email_address = (known after apply)
                  + id            = (known after apply)
                  + type          = (known after apply)
                  + uri           = (known after apply)
                }
            }

          + owner {
              + display_name = (known after apply)
              + id           = (known after apply)
            }
        }
    }

Plan: 4 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_s3_bucket.firehose_events: Creating...
aws_s3_bucket.firehose_events: Creation complete after 1s [id=pinpoint-error]
aws_s3_bucket_acl.bucket_acl: Creating...
aws_kinesis_firehose_delivery_stream.fh_stream: Creating...
aws_s3_bucket_acl.bucket_acl: Creation complete after 0s [id=pinpoint-error,private]
aws_kinesis_firehose_delivery_stream.fh_stream: Still creating... [10s elapsed]
aws_kinesis_firehose_delivery_stream.fh_stream: Still creating... [20s elapsed]
aws_kinesis_firehose_delivery_stream.fh_stream: Still creating... [30s elapsed]
aws_kinesis_firehose_delivery_stream.fh_stream: Still creating... [40s elapsed]
aws_kinesis_firehose_delivery_stream.fh_stream: Still creating... [50s elapsed]
aws_kinesis_firehose_delivery_stream.fh_stream: Still creating... [1m0s elapsed]
aws_kinesis_firehose_delivery_stream.fh_stream: Still creating... [1m10s elapsed]
aws_kinesis_firehose_delivery_stream.fh_stream: Still creating... [1m20s elapsed]
aws_kinesis_firehose_delivery_stream.fh_stream: Still creating... [1m30s elapsed]
aws_kinesis_firehose_delivery_stream.fh_stream: Creation complete after 1m34s [id=arn:aws:firehose:us-east-1:538069693173:deliverystream/firehose]
aws_pinpoint_event_stream.pinpoint_firehose: Creating...
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [10s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [20s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [30s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [40s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [50s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [1m0s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [1m10s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [1m20s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [1m30s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [1m40s elapsed]
aws_pinpoint_event_stream.pinpoint_firehose: Still creating... [1m50s elapsed]
╷
│ Error: error putting Pinpoint Event Stream for application xxxxxxxxxxxxxxxxxxxxx: BadRequestException: Destination stream validation failed. Check Destination and Role ARN values and make sure the IAM Role is configured correctly.
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "4801e8d1-82a0-43b6-87a7-f41c76a15e5f"
│   },
│   Message_: "Destination stream validation failed. Check Destination and Role ARN values and make sure the IAM Role is configured correctly.",
│   RequestID_: "4801e8d1-82a0-43b6-87a7-f41c76a15e5f"
│ }
│
│   with aws_pinpoint_event_stream.pinpoint_firehose,
│   on main.tf line 80, in resource "aws_pinpoint_event_stream" "pinpoint_firehose":
│   80: resource "aws_pinpoint_event_stream" "pinpoint_firehose" {
│
╵

Panic Output

│ Error: error putting Pinpoint Event Stream for application 9398a64865d24c4d9e69e39d5eab8bd9: BadRequestException: Destination stream validation failed. Check Destination and Role ARN values and make sure the IAM Role is configured correctly.
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "4801e8d1-82a0-43b6-87a7-f41c76a15e5f"
│   },
│   Message_: "Destination stream validation failed. Check Destination and Role ARN values and make sure the IAM Role is configured correctly.",
│   RequestID_: "4801e8d1-82a0-43b6-87a7-f41c76a15e5f"
│ }
│
│   with aws_pinpoint_event_stream.pinpoint_firehose,
│   on main.tf line 80, in resource "aws_pinpoint_event_stream" "pinpoint_firehose":
│   80: resource "aws_pinpoint_event_stream" "pinpoint_firehose" {

Important Factoids

No response

References

• https://github.com/hashicorp/terraform-provider-aws/issues/15899
• https://docs.aws.amazon.com/pinpoint/latest/developerguide/permissions-streams.html#permissions-streams-permissionspolicies

Would you like to implement a fix?

No response

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[loganpowell]

Upon further investigation, I've gotten terraform to successfully deploy after making this change

resource "aws_iam_role" "firehose_role" {
  name = "iam_firehose_role"
  assume_role_policy = jsonencode({
    Version : "2012-10-17",
    Statement : [
      {
        Effect : "Allow",
        Action : "sts:AssumeRole",
        Principal : {
-          Service : "firehose.amazonaws.com"
+          Service : "pinpoint.us-east-1.amazonaws.com"
        }
      }
    ]
  })
}

However, looking at the console, the IAM role hasn't been applied

[loganpowell]

Ok, got it working with this setup:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.32.0"
    }
  }
}

provider "aws" {
  alias   = "email"
  region  = "us-east-1"
  profile = "default"
}

resource "aws_iam_role" "pinpoint_role" {
  name = "iam_firehose_role"
  assume_role_policy = jsonencode({
    Version : "2012-10-17",
    Statement : [
      {
        Effect : "Allow",
        Action : "sts:AssumeRole",
        Principal : {
          Service : "pinpoint.amazonaws.com"
        },
        Sid : ""
      }
    ]
  })
}
resource "aws_iam_role_policy" "firehose_policy" {
  name = "pinpoint_firehose_policy"
  role = aws_iam_role.pinpoint_role.id
  policy = jsonencode({
    Version : "2012-10-17",
    Statement : [{
      Effect : "Allow",
      Action : [
        "firehose:PutRecordBatch",
        "firehose:DescribeDeliveryStream"
      ],
      Resource : [
        "arn:aws:firehose:us-east-1:*:*/*"
      ]
    }]
  })
}

resource "aws_s3_bucket" "firehose_events" {
  bucket = "pinpoint-error"
}
resource "aws_s3_bucket_acl" "bucket_acl" {
  bucket = aws_s3_bucket.firehose_events.id
  acl    = "private"
}
resource "aws_iam_role" "firehose_role" {
  name = "firehose_test_role"
  assume_role_policy = jsonencode({
    Version : "2012-10-17",
    Statement : [
      {
        Effect : "Allow",
        Action : "sts:AssumeRole",
        Principal : {
          Service : "firehose.amazonaws.com"
        },
        Sid : ""
      }
    ]
  })
}
# Grant FH Role Permissions to S3
resource "aws_iam_role_policy" "s3_policy" {
  name = "s3_fh_policy"
  role = aws_iam_role.firehose_role.id
  policy = jsonencode({
    Version : "2012-10-17",
    Statement : [{
      Effect : "Allow",
      Action : [
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:PutObject"
      ],
      Resource : [
        "arn:aws:s3:::${aws_s3_bucket.firehose_events.id}",
        "arn:aws:s3:::${aws_s3_bucket.firehose_events.id}/*"
      ]
    }]
  })
}

resource "aws_kinesis_firehose_delivery_stream" "fh_destination" {
  name        = "firehose"
  destination = "extended_s3"
  extended_s3_configuration {
    role_arn            = aws_iam_role.firehose_role.arn
    bucket_arn          = aws_s3_bucket.firehose_events.arn
    prefix              = "!{timestamp:yyyy/MM/dd}/"
    error_output_prefix = "!{timestamp:yyyy/MM/dd}/!{firehose:error-output-type}"
    buffer_interval     = 360 # range: 60 to 900 seconds
    buffer_size         = 64  # range: 64 to 128 MiB
  }
}

resource "aws_pinpoint_app" "my_app" {
  name = "elvis_lives"
}
resource "aws_pinpoint_event_stream" "pinpoint_firehose" {
  application_id         = aws_pinpoint_app.my_app.id
  destination_stream_arn = aws_kinesis_firehose_delivery_stream.fh_destination.arn
  role_arn               = aws_iam_role.pinpoint_role.arn
}

It might be good to add this to the docs as an example and save others some time.

Thank you for terraform! 🙏

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.