Error: 429 Too Many Requests

[fdmota]

I get the following errors:

│ Error: 429 Too Many Requests
│ 
│   with module.sentry_kafka_topics.confluentcloud_kafka_topic.kafka_topics["outcomes"],
│   on ../../../../../../modules/confluent-kafka-topics/main.tf line 1, in resource "confluentcloud_kafka_topic" "kafka_topics":
│    1: resource "confluentcloud_kafka_topic" "kafka_topics" {

with code that looks like this:

resource "confluentcloud_kafka_topic" "kafka_topics" {
  for_each = var.topics

  topic_name       = each.value.topic_name
  partitions_count = each.value.partitions_count
  config           = each.value.config

  kafka_cluster = var.cluster_info.cluster_id
  http_endpoint = var.cluster_info.http_endpoint
  credentials {
    key    = local.cluster_credentials.api
    secret = local.cluster_credentials.secret
  }
}

I temporarily resolved the issue by adding a sleep of 1 second between calls in the provider

func kafkaTopicRead(ctx context.Context, d *schema.ResourceData, meta interface{}) diag.Diagnostics {
        log.Printf("[INFO] Kafka topic read2 for %s", d.Id())
        throttle() <-- Sleep 1s

        clusterId := extractClusterId(d)
        topicName := extractTopicName(d)

Is there a better solution?

[linouk23]

Thanks for opening an issue!

Could you try https://github.com/confluentinc/terraform-provider-confluentcloud/issues/15#issuecomment-972131964?

[fdmota]

It works but it feels like a workaround... Any plans to change this?

[linouk23]

Any plans to change this?

We're anticipating a rate limit increase for a corresponding API as soon as TF provider goes into Preview / General Access from the current Early Access stage. Alternatively, we might be able to merge something similar to your smart fix which should work as well.

[linouk23]

@fdmota could you share more details about your fix with throttle()? It might be a good idea for us to implement something similar while waiting for a rate limit increase.

FYI in order to fix the issue we're going to implement a retry logic for 429 errors.

[fdmota]

it was nothing very fancy. I created a quick sleep of 1 second to experiment if it would work.

func throttle () {
        time.Sleep(1 * time.Second)
        log.Printf("[INFO] Sleep Over.....")
}

[linouk23]

👋 @fdmota @vilmosnagy please check out our most recent 0.4.0 release where we fixed the issue!