Hey Have a simple question i think. I am using a Broadway Kafka pipeline with
producer -> 1
processors -> 10
batcher -> 10
Seen in the image attached.
My questions is just i was wondering if my assumption here about how this is working is correct
10 processors running
each ones job is to grab 1 message from Kafka, execute whatever logic is within the handle_message callback and send to batcher as long as i am saying Message.put_batcher(:name_of_batcher) which i am
1 batcher running
this ones job is just to aggregate each message sent to it and when it reaches a threshold of batch_size then it will send the batch off to its downstream processors
10 batcher processors running
each ones job is just to take each batch of messages it gets and execute the logic for handle_batch
josevalim
commented
2 years ago
Hey Have a simple question i think. I am using a Broadway Kafka pipeline with
Seen in the image attached.
My questions is just i was wondering if my assumption here about how this is working is correct
10 processors running each ones job is to grab 1 message from Kafka, execute whatever logic is within the handle_message callback and send to batcher as long as i am saying Message.put_batcher(:name_of_batcher) which i am
1 batcher running this ones job is just to aggregate each message sent to it and when it reaches a threshold of batch_size then it will send the batch off to its downstream processors
10 batcher processors running each ones job is just to take each batch of messages it gets and execute the logic for handle_batch