search

apache / accumulo

Apache Accumulo
https://accumulo.apache.org
Apache License 2.0
1.05k stars 444 forks source link

Investigate idle compactor load on zookeeper #4666

Closed ddanielr closed 6 days ago

ddanielr commented 1 month ago

When running some external compactor tests, it was noticed that zk resource usage seemed to increase in relation to the number of idle compactors. This did not seem to occur if the compactors were busy.

This should be investigated to ensure we aren't creating additional ZK load as part of the job polling operations.

General Investigation Steps:

  1. Start a running accumulo instance
  2. Check ZK usage (watchers, etc) https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#sc_zkCommands
  3. Create a external compaction service
  4. Trigger a user compaction for a compaction queue that would queue up work
  5. Spin up external compactors and ensure they are busy (compactor's busy metric).
  6. Monitor ZK usage
  7. Either cancel the external compactions or spin down existing compactors and create new ones with no work.
  8. Monitor ZK usage
keith-turner commented 1 month ago

Its possible that #4681 was a cause of this, but not sure.

ctubbsii commented 2 weeks ago

@ddanielr Does this still occur after #4681 ?

ddanielr commented 6 days ago

Unable to replicate this problem after #4681 was merged. Closing issue