Open gauravAshok opened 5 months ago
How do we handle Consumer failure scenarios and prevent duplicate delivery during recovery?
Eg: Consumer task fails post ConcurrencyControl::deliver
for some of the messages. So, the offset is not committed to the messaging stack. Now, when the consumer comes back up, or the workload is assigned to another consumer. It will try to deliver those messages again. Are we fine with that?
Consumer task? (ctx(event_loop))
todo: explore if post_process needs to be batched!, everything before the post_process is batched upto some extent. post_process is the only step that is per msg. enqueue from outside onto the ctx, is also a candidate for contention. Should the event-loop be responsible for http/remote requests? probably not, as one is highly IO bound and other is compute bound. IO threads need to be responsive.