apache / accumulo

Apache Accumulo
https://accumulo.apache.org
Apache License 2.0
1.07k stars 444 forks source link

Move monitor aggregation point from Manager to Monitor #4973

Open dlmarion opened 3 hours ago

dlmarion commented 3 hours ago

Is your feature request related to a problem? Please describe. The Monitor could potentially become an issue in 4.0 if we end up needing to implement multiple active managers. Multiple Managers may be needed in 4.0 due to the increased workload of the Manager, which has assumed all Tablet maintenance operations. The issue here is not with the Monitor, but that the Manager is the aggregation point for most or all of the information that the Monitor uses.

Describe the solution you'd like I'm wondering if we can change the aggregation point from the Manager to the Monitor itself. Here's what I'm thinking:

  1. Create a custom MeterRegistry and wire it up to the Micrometer global registry. This custom MeterRegistry would just keep the most recent metrics being reported in the server (not performing any aggregation).
  2. Create and start a Jetty server in each server process (sans the Monitor). Register the host/port as a new Service in the server ServiceLock.
  3. Modify the Monitor to hit the http endpoints for the servers listed in ZooKeeper and aggregate the resulting information. The http endpoint would return the latest micrometer metrics, any other values not being recorded as metrics being used by the monitor, and potentially log messages (we could maybe get rid of the logging appender?). I believe that with the HttpClient in Java11 and using Jetty, this can be done using http/2 in an asynchronous manner.
  4. Remove the monitor related code and endpoints from the Manager (ManagerMonitorInfo, etc.).

Describe alternatives you've considered Removing the Monitor in favor of just using industry standard monitoring and log aggregation solutions

dlmarion commented 3 hours ago

Question: The Jetty server in the Monitor does not support Kerberos and does not seem subject to that requirement. Would the Jetty servers included in this proposal be subject to that requirement?