search

linkedin / brooklin

An extensible distributed system for reliable nearline data streaming at scale
BSD 2-Clause "Simplified" License
922 stars 137 forks source link

Added log messages to debug assignPartitions #951

Closed jzakaryan closed 1 year ago

jzakaryan commented 1 year ago

We observed that the leader has a tendency to get stuck on this method (in some rare cases for more than 5 minutes). Added log messages to help debug the issue.

Note that the test suite in TestLoadBasedPartitionAssigner do not capture such performance problems. Attempts to reproduce them locally didn't work.

shrinandthakkar commented 1 year ago

I remember us talking about putting logs around any IO operations that might be happening, should we add some logs there as well?

jzakaryan commented 1 year ago

I remember us talking about putting logs around any IO operations that might be happening, should we add some logs there as well?

The consensus at the end of the meeting was that the assignPartitions method is where the CPU time is spent. I avoided adding logs in other places as to keep unnecessary logging to minimum.