search

robinhood / faust

Python Stream Processing
Other
6.7k stars 538 forks source link

Faust Application processes messages but does not commit offset after connection to kafka broker is reset #736

Open vinaygopalkrishnan opened 2 years ago

vinaygopalkrishnan commented 2 years ago

I see this issue when the connection to the kafka broker is reset. In such a case, the faust application continues to process messages, but is not able to commit offset with the following error:

Could not send <class 'aiokafka.protocol.transaction.AddOffsetsToTxnRequest_v0'>: StaleMetadata('Broker id 8 not in current metadata')

This occurs till the faust application restarts.

Once the application restarts, it re-processes messages and starts committing offsets

As a result, a number of duplicate messages are processed.

bobh66 commented 2 years ago

This project appears to have been abandoned.

You might want to check out the fork of this project - https://github.com/faust-streaming/faust