[Bug]: aws_kinesis_stream doesn't error out on reaching the limit number of streams for the account

[vainkop]

Terraform Core Version

1.8.5

AWS Provider Version

5.55.0

Affected Resource(s)

aws_kinesis_stream

Expected Behavior

An error should be printed & Terraform should exit with exit code 1.

Th AWS console error looks like so:

This request would exceed the limit on the number of On-Demand streams for the account xxx in region_name. Current On-Demand streams count: XX. Limit: XX.

Actual Behavior

Terraform seems to keep trying to create the resources (1 hour by default) while the AWS console immediately errors out with the following error:

This request would exceed the limit on the number of On-Demand streams for the account xxx in region_name. Current On-Demand streams count: XX. Limit: XX.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

resource "aws_kinesis_stream" "stream" {
  name                      = var.namevar.name]))
  shard_count               = var.stream_mode == "ON_DEMAND" ? null : var.shard_count
  retention_period          = var.retention_period
  enforce_consumer_deletion = var.enforce_consumer_deletion # true by default in my environment

  stream_mode_details {
    stream_mode = var.stream_mode
  }

  shard_level_metrics = var.shard_level_metrics
}

Steps to Reproduce

Try to create more kinesis streams than available per account service limits.

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

No

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[vainkop]

any updates? seeing up to 60 minutes of deployment & failing only in the end really is annoying, takes too much time & dissapoints. other resources are implemented better if they limits they report of them immediately & here there's a limit which can even aws cli can poll from the api so it should be there.

module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [43m50s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [43m50s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [44m0s elapsed]
module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [44m0s elapsed]
module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [44m10s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [44m10s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [44m20s elapsed]
module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [44m20s elapsed]
module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [44m30s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [44m30s elapsed]
module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [44m40s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [44m40s elapsed]
module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [44m50s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [44m50s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [45m0s elapsed]
module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [45m0s elapsed]
module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [45m10s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [45m10s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [45m20s elapsed]
module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [45m20s elapsed]
module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [45m30s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [45m30s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [45m40s elapsed]
module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [45m40s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [45m50s elapsed]
module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [45m50s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [46m0s elapsed]
module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [46m0s elapsed]
module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [46m10s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [46m10s elapsed]
module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [46m20s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [46m20s elapsed]
module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [46m30s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [46m30s elapsed]
module.kinesis_abc_def.aws_kinesis_stream.stream: Still creating... [46m40s elapsed]
module.kinesis_ghi_log.aws_kinesis_stream.stream: Still creating... [46m40s elapsed]
...

[vainkop]

In the meantime here's a workaround for those who might need it:

resource "null_resource" "check_aws_limits" {
  provisioner "local-exec" {
    command = "python3 ${path.module}/check_aws_limits.py"
  }
}

resource "aws_kinesis_stream" "stream" {
...
  depends_on = [null_resource.check_aws_limits]
}

# check_aws_limits.py
import boto3
import json
import sys

def check_kinesis_limits():
    client = boto3.client('kinesis')

    # Fetch Kinesis limits
    limits = client.describe_limits()

    # Extract limit values
    on_demand_stream_count = limits['OnDemandStreamCount']
    on_demand_stream_count_limit = limits['OnDemandStreamCountLimit']

    # Print the extracted values
    print(f"OnDemandStreamCount={on_demand_stream_count}")
    print(f"OnDemandStreamCountLimit={on_demand_stream_count_limit}")

    # Check limits and exit with appropriate code
    if on_demand_stream_count >= on_demand_stream_count_limit:
        error_message = (
        "OnDemandStreamCountLimit is less or equal to OnDemandStreamCount!\n"
        "#######################################################\n"
        "#######################################################\n"
        "###      AWS LIMITS ARE REACHED !!!                 ###\n"
        "###      REQUEST MORE STREAMS FROM AWS SUPPORT      ###\n"
        "#######################################################\n"
        "#######################################################\n"
        )
        print(error_message)
        sys.exit(1)

    print("Kinesis limits are within acceptable range.")
    sys.exit(0)

if __name__ == "__main__":
    check_kinesis_limits()

Output:

│ Error: local-exec provisioner error
│ 
│   with module.kinesis_abc.null_resource.check_aws_limits,
│   on .terraform/modules/kinesis/main.tf line 17, in resource "null_resource" "check_aws_limits":
│   17:   provisioner "local-exec" {
│ 
│ Error running command 'python3
│ .terraform/modules/kinesis/check_aws_limits.py':
│ exit status 1. Output: OnDemandStreamCount=50
│ OnDemandStreamCountLimit=50
│ OnDemandStreamCountLimit is less or equal to OnDemandStreamCount!
│ #######################################################
│ #######################################################
│ ###      AWS LIMITS ARE REACHED !!!                 ###
│ ###      REQUEST MORE STREAMS FROM AWS SUPPORT      ###
│ #######################################################
│ #######################################################
│ 
│ 
╵
╷
│ Error: local-exec provisioner error
│ 
│   with module.kinesis_def.null_resource.check_aws_limits,
│   on .terraform/modules/kinesis_def/kinesis/main.tf line 17, in resource "null_resource" "check_aws_limits":
│   17:   provisioner "local-exec" {
│ 
│ Error running command 'python3
│ .terraform/modules/kinesis_def/kinesis/check_aws_limits.py':
│ exit status 1. Output: OnDemandStreamCount=50
│ OnDemandStreamCountLimit=50
│ OnDemandStreamCountLimit is less or equal to OnDemandStreamCount!
│ #######################################################
│ #######################################################
│ ###      AWS LIMITS ARE REACHED !!!                 ###
│ ###      REQUEST MORE STREAMS FROM AWS SUPPORT      ###
│ #######################################################
│ #######################################################