Unable to pull topics using Terraform

maxverbosity commented 9 months ago

Summary

When running terraform plan, the refresh fails and errors out on topics, claiming too may connections.

kafka.tf snippet

resource "upstash_kafka_topic" "topic" {
    for_each = { for topic in var.cluster_topics : topic.topic_name => topic }

    topic_name       = "${ each.key }"
    partitions       = "${ each.value.partitions }"
    retention_time   = "${ each.value.retention_time }"
    retention_size   = "${ each.value.retention_size }"
    max_message_size = "${ each.value.max_message_size }"
    cleanup_policy   = "${ each.value.cleanup_policy }"
    cluster_id       = upstash_kafka_cluster.cluster.cluster_id
}

Terraform output snippet

<snip>
Planning failed. OpenTofu encountered an error while generating this plan.

╷
│ Error: Get Kafka Topic failed, status code: 429 response: <html>
│ <head><title>429 Too Many Requests</title></head>
│ <body>
│ <center><h1>429 Too Many Requests</h1></center>
│ <hr><center>nginx</center>
│ </body>
│ </html>
│
│
│   with module.v1.module.prod_cluster.upstash_kafka_topic.topic["ccm.lateralled"],
│   on ../../../modules/digitalocean/kafka/1.0.0/main.tf line 17, in resource "upstash_kafka_topic" "topic":
│   17: resource "upstash_kafka_topic" "topic" {
│
╵
╷
│ Error: Get Kafka Topic failed, status code: 429 response: <html>
│ <head><title>429 Too Many Requests</title></head>
│ <body>
│ <center><h1>429 Too Many Requests</h1></center>
│ <hr><center>nginx</center>
│ </body>
│ </html>
│
│
│   with module.v1.module.prod_cluster.upstash_kafka_topic.topic["registrant.class"],
│   on ../../../modules/digitalocean/kafka/1.0.0/main.tf line 17, in resource "upstash_kafka_topic" "topic":
│   17: resource "upstash_kafka_topic" "topic" {
│
╵
╷
│ Error: Get Kafka Topic failed, status code: 429 response: <html>
│ <head><title>429 Too Many Requests</title></head>
│ <body>
│ <center><h1>429 Too Many Requests</h1></center>
│ <hr><center>nginx</center>
│ </body>
│ </html>
│
│
│   with module.v1.module.prod_cluster.upstash_kafka_topic.topic["event.updated"],
│   on ../../../modules/digitalocean/kafka/1.0.0/main.tf line 17, in resource "upstash_kafka_topic" "topic":
│   17: resource "upstash_kafka_topic" "topic" {
│
╵
</snip>

Expected outcome

The plan should succeed.

maxverbosity commented 9 months ago

I've figured out the cause of the problem.

For background, we have 68 topics we are creating in this loop. Terraform is running those resource creations in parallel which is being flagged on your end and throttled.

Adding a flag to the execution fixes the problem. Removing it causes the problem to occur again.

# This works
$ terraform plan -parallelism=1

# This does NOT work and I get the 429 errors listed above
$ terraform plan

I'm not sure what else we can do on our end other than use the flag. We can work around this with the parallelism flag but it doesn't seem like a viable permanent solution. I don't think our number of topics is large and we don't have insight into your systems to know what's happening when the request hits.

maxverbosity commented 9 months ago

Is any one even monitoring this? I submitted this ticket 4 days ago and haven't received anything- not even an acknowledgement that the issue has been received or read.

fahreddinozcan commented 9 months ago

Hey,

sorry for the latency here. I will be investigating this issue, and see what I can come up with

maxverbosity commented 9 months ago

No worries- thanks for looking into it.

fahreddinozcan commented 9 months ago

Hey, I wanted to update you on the progress here. I've looked into several possible solutions, however none were feasible in this case. Unfortunately, Terraform doesn't support batch requests other than minimal internal optimizations. So we can't really intercept and batch the requests as far as I see. One other solution could be to utilize caching and reduce the API call count, however Terraform doesn't have support for that either. For both of these topics, there's an ongoing issue on the sdk plugin repo.

We've also considered increasing the request limit per second on the API, however it doesn't really seem possible to scale it to these numbers, and we'd still need to update the limit whenever someone hits this limit on the Terraform. So we should be able to solve this problem on the TF Provider side.

I will keep looking into this, and update you as soon as I find a feasible solution.

binnsr commented 5 months ago

Is there an update on this issue?

fahreddinozcan commented 4 months ago

@binnsr unfortunately not. I will check with the Terraform SDK maintainers, to see if they have any work related to parallelism.

upstash / terraform-provider-upstash