Running into throttling issues with many firehose deployments

[ghost]

This issue was originally opened by @ppeterson-evotek as hashicorp/terraform#23598. It was migrated here as a result of the provider split. The original body of the issue is below.

---

* aws_kinesis_firehose_delivery_stream.kinesis_stream_es_nginx_access_admin: Error Updating Kinesis Firehose Delivery Stream: "firehose-nginx-access-admin"
InvalidArgumentException: Received ThrottlingException while describing the ElasticSearch domain.
    status code: 400, request id: fdce4a2e-c880-c71f-abaf-34dcdfeff05c
* module.elasticsearch_domain.aws_kinesis_firehose_delivery_stream.kinesis_stream_es_nginx_access_api: 1 error(s) occurred:

* aws_kinesis_firehose_delivery_stream.kinesis_stream_es_nginx_access_api: Error Updating Kinesis Firehose Delivery Stream: "firehose-nginx-access-api"
InvalidArgumentException: Received ThrottlingException while describing the ElasticSearch domain.
    status code: 400, request id: c7864577-ebed-e197-91e7-3b805a6478b5
* module.elasticsearch_domain.aws_kinesis_firehose_delivery_stream.kinesis_stream_es_nginx_access_webhooks: 1 error(s) occurred:

* aws_kinesis_firehose_delivery_stream.kinesis_stream_es_nginx_access_webhooks: Error Updating Kinesis Firehose Delivery Stream: "firehose-nginx-access-webhooks"
InvalidArgumentException: Received ThrottlingException while describing the ElasticSearch domain.
    status code: 400, request id: fbfc1ac7-827e-9a4a-ad9d-6430a3e25340
* module.elasticsearch_domain.aws_kinesis_firehose_delivery_stream.kinesis_stream_es_nginx_access_consumer-out: 1 error(s) occurred:

* aws_kinesis_firehose_delivery_stream.kinesis_stream_es_nginx_access_consumer-out: Error Updating Kinesis Firehose Delivery Stream: "firehose-nginx-access-consumer-out"
InvalidArgumentException: Received ThrottlingException while describing the ElasticSearch domain.
    status code: 400, request id: fe5a8875-9b7c-0377-a83b-f68fdcb325bc
* module.elasticsearch_domain.aws_kinesis_firehose_delivery_stream.kinesis_stream_es_nginx_error_consumer-out: 1 error(s) occurred:```

# Expected Behavior
All 60 kinesis streams should have been updated

### Actual Behavior
Api throttling stopped the deployment

### Steps to Reproduce
1. Create 60+ kinesis firehose streams in terraform
2. Apply

### Additional Context
I contacted amazon to see if there was some way to increase the throttling threshhold, but I doubt this is an option, so putting in some kind of delay on terraform's side would i guess be what i expect

[justinretzolk]

Hey @ppeterson-evotek 👋 Thank you for taking the time to file this issue. Given that there's been a number of AWS provider releases since you initially filed it, can you confirm whether you're still experiencing this behavior?

[ljluestc]

provider "aws" {
  region = "us-east-1" # Replace with your desired region
}

# List of Kinesis Firehose stream names
variable "firehose_streams" {
  type    = "list"
  default = [
    "firehose-nginx-access-admin",
    "firehose-nginx-access-api",
    "firehose-nginx-access-webhooks",
    "firehose-nginx-access-consumer-out",
    "firehose-nginx-error-consumer-out"
    # Add more streams as needed
  ]
}

# Create Kinesis Firehose delivery streams
resource "aws_kinesis_firehose_delivery_stream" "kinesis_stream" {
  for_each = toset(var.firehose_streams)

  name        = each.value
  destination = "elasticsearch" # Specify your destination

  elasticsearch_configuration {
    # Add your Elasticsearch configuration here
  }
}

# Null resource to introduce delay
resource "null_resource" "throttle_delay" {
  count = length(var.firehose_streams)

  provisioner "local-exec" {
    command = "sleep 10" # Adjust the delay as needed
  }

  depends_on = [
    aws_kinesis_firehose_delivery_stream.kinesis_stream
  ]
}

output "firehose_stream_names" {
  value = aws_kinesis_firehose_delivery_stream.kinesis_stream.*.name
}