This PR is built upon #6683 and #9172 and aims to reduce the number of ZooKeeper watch counts.
Fixed Huge Number of Watches in ZooKeeper
Previously, Announcer.java causes all child nodes under the parent path to be watched by the ZooKeeper ensemble. This causes an unnecessarily large number of ZooKeeper watches to be produced.
The new NodeAnnouncer.java class, which is heavily modelled on the Announcer.java, aims to address this issue by announcing a single node within a ZooKeeper ensemble. By eliminating the watches on child nodes, this approach significantly reduces the total number of watch counts in ZooKeeper. Tests conducted on the production server also indicate a decrease in watch counts resulting from this change.
The original Announcer.java still provides better performance for segment announcements, and hence will be retained in the codebase.
Documentation
Implementation of #6683 and #9172 are complete, and this PR addresses blocking requests for change by adding Javadocs, and making announcement logic more human-readable.
Shift Announceable class in Announcer.java to Announceable.java.
Refactor long methods by creating helper functions.
Add ZKPathsUtils.java to abstract the retrieval of ZooKeeper path and ZooKeeper node.
Release note
Improved: ZooKeeper no longer spins up an unnecessary large number of watches when running realtime tasks.
Key changed/added classes in this PR
Announcer.java
NodeAnnouncer.java
Announceable.java
AnnouncerModule.java
ZKPathsUtils.java
Further Actionable
I plan to create issues for the following follow-up actions after this PR:
Deprecated Code
The PathChildrenCache, NodeCache classes have been deprecated since Curator 5.0.0.
We can look into replacing these deprecated classes with CuratorCache. CuratorCache requires ZooKeeper 3.6+, and we are currently using ZooKeeper 3.8.4.
Concurrency Flow
Add a concurrent control flow documentation for NodeAnnouncer.
This PR has:
[x] been self-reviewed.
[x] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
[x] a release note entry in the PR description.
[x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[x] added or updated version, license, or notice information in licenses.yaml
[x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
Fixes #6647
Description
This PR is built upon #6683 and #9172 and aims to reduce the number of ZooKeeper watch counts.
Fixed Huge Number of Watches in ZooKeeper
Previously,
Announcer.javacauses all child nodes under the parent path to be watched by the ZooKeeper ensemble. This causes an unnecessarily large number of ZooKeeper watches to be produced.The new
NodeAnnouncer.javaclass, which is heavily modelled on theAnnouncer.java, aims to address this issue by announcing a single node within a ZooKeeper ensemble. By eliminating the watches on child nodes, this approach significantly reduces the total number of watch counts in ZooKeeper. Tests conducted on the production server also indicate a decrease in watch counts resulting from this change.The original
Announcer.javastill provides better performance for segment announcements, and hence will be retained in the codebase.Documentation
Refactoring
Announceableclass inAnnouncer.javatoAnnounceable.java.ZKPathsUtils.javato abstract the retrieval of ZooKeeper path and ZooKeeper node.Release note
Improved: ZooKeeper no longer spins up an unnecessary large number of watches when running realtime tasks.
Key changed/added classes in this PR
Announcer.javaNodeAnnouncer.javaAnnounceable.javaAnnouncerModule.javaZKPathsUtils.javaFurther Actionable
I plan to create issues for the following follow-up actions after this PR:
Deprecated Code
The
PathChildrenCache,NodeCacheclasses have been deprecated since Curator 5.0.0.We can look into replacing these deprecated classes with
CuratorCache. CuratorCache requires ZooKeeper 3.6+, and we are currently using ZooKeeper 3.8.4.Concurrency Flow
Add a concurrent control flow documentation for NodeAnnouncer.
This PR has: