DataDog / kafka-kit

Kafka storage rebalancing, automated replication throttle, cluster API and more
Apache License 2.0
488 stars 54 forks source link

[registry] CreateTopic tagging retries #414

Closed jamiealquiza closed 2 years ago

jamiealquiza commented 2 years ago

The current CreateTopic rpc can optionally take tags. When provided, the tagging operation runs immediately after the request to create the topic in Kafka, which can show lag at time, especially with large partition counts.

This change adds a backoff retry mechanism to improve the creation of tagged topics.

Before:

 % curl -XPOST $(docker port kafka-kit_registry_1 8080)/v1/topics/create -d '{"topic": {"name": "test8", "partitions": 500, "replication": 2, "tags": {"key":"value"}}}'
{"code":5, "message":"topic does not exist", "details":[]}

After:

% curl -XPOST $(docker port kafka-kit_registry_1 8080)/v1/topics/create -d '{"topic": {"name": "test9", "partitions": 500, "replication": 2, "tags": {"key":"value"}}}'
{}
jamiealquiza commented 2 years ago

(merging to prevent staleness; entire branch is a wip/breaking change effort anyway)