[watermill-kafka] "read tcp <ip>:<port>: i/o timeout" when kafka rebalances

ThreeDotsLabs / watermill

Building event-driven applications the easy way in Go.

https://watermill.io

MIT License

7.75k stars 406 forks source link

[watermill-kafka] "read tcp <ip>:<port>: i/o timeout" when kafka rebalances #376

Open zaidoon1 opened 1 year ago

zaidoon1 commented 1 year ago

This appears to be the same issue as https://github.com/Shopify/sarama/issues/1192? And it can be fixed by doing something like https://github.com/IBM/sarama/commit/0151486eeb29f538508f313a3c4f9c1c11ee5e04 ?

AlexCuse commented 1 year ago

This seems like it could help. Using select vs ranging the channel here feels like the right thing to do too. I am not sure I know kafka well enough to reproduce reliably but can make the change and open a PR if tests pass.

AlexCuse commented 1 year ago

Tried making this change and seeing some failures in tests - appear to be related to nil messages coming through the channel.