CallistoLabsNYC / samsa

Rust-native Kafka/Redpanda protocol and client implementation.
Apache License 2.0
106 stars 5 forks source link

Round Robin Assignment for Producers #102

Open nathan-brainkey opened 1 month ago

nathan-brainkey commented 1 month ago

Is there a way to use round robin assignment to partitions for producers?

From the example:

let stream = iter(0..5000).map(|_| ProduceMessage {
    topic: topic_name.to_string(),
    partition_id,
    key: Some(bytes::Bytes::from_static(b"Tester")),
    value: Some(bytes::Bytes::from_static(b"Value")),
    headers: vec![
        samsa::prelude::Header::new(String::from("Key"), bytes::Bytes::from("Value"))
    ],
}).chunks(100);

It doesn't look like partition_id is Optional.

New to Samsa and just wondering how I might do this. Thanks for the help!

morukele commented 3 weeks ago

Hi @nathan-brainkey, yes, the partition_id is mandatory. However, you can do a round-robin based on the number of partitions. Here is an example code:

let stream = iter(0..NUMBER_OF_PARTITIONS).map(|i| ProduceMessage {
        topic: topic_name.to_string(),
        partition_id: i,
        key: None,
        value: Some(bytes::Bytes::from_static(b"0123456789")),
        headers: vec![],
    });

You can find more of this example in the project's tests/multi_partitions_setup.rs file.

Let me know if this helps you.