kafka-franz: Add more configurable fields for kafka-franz performance tuning - Githubissues

warpstreamlabs / bento

Fancy stream processing made operationally mundane. This repository is a fork of the original project before the license was changed.

https://warpstreamlabs.github.io/bento/

Other

1.06k stars 71 forks source link

kafka-franz: Add more configurable fields for kafka-franz performance tuning #129

Closed gregfurman closed 1 month ago

gregfurman commented 1 month ago

Adds the following fields to input.kafka_franz:

A list of group_balancers (default cooperative_sticky)
metadata_max_age (default 5m)
fetch_max_bytes (default 50MiB)
fetch_max_partition_bytes (default 1MiB)
fetch_max_wait (default 5s)
preferring_lag (optional, no default)

Adds the following fields to output.kafka_franz:

partitioner has a uniform_bytes enum
uniform_bytes_options (default { "bytes" : "1MiB", "adaptive" : false, "keys" : false })
- Note: we do not yet allow specifying a PartitionerHasher. This approach should probably be reworked using bloblang with some lint rule.
max_buffered_records (default 10K)
metadata_max_age (default 5m)

A new integration test TestIntegrationKafkaWarpstreamDefaults has also been added with the performance tuning defaults found here.