feat: mute collectors based on exception count

[kartva]

Is your feature request related to a problem? Please provide an existing Issue # , or describe. As discussed at https://github.com/opensearch-project/performance-analyzer/issues/549

Describe the solution you are proposing We use the exception counts maintained by StatsCollector to dynamically mute collectors based on the number of exceptions they throw in the last minute of sampling.

Additional context Currently in the proof of concept stage. If this is successful, further avenues of investigation may be:

• tracking the number of bytes written by each collector and using that data to dynamically control collectors
• collecting more nuanced performance information about collectors such as their CPU consumption

Check List

• [ ] New functionality includes testing.
  • [ ] All tests pass
• [ ] New functionality has been documented.
  • [ ] New functionality has javadoc added
• [ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[kartva]

What sort of configuration setting do you think would be appropriate for exceptionMuteLimit? Would this have a default value (like 5) or be off by default? I imagine having it off by default will be sensible.

[kartva]

summary of changes so far:

• added exception counting mechanism to mute collectors
• changed invokePrivileged to not catch errors, since exception counting relies on the exceptions not being caught and logged earlier in the chain. invokePrivilegedAndLogError is more appropriate for when exceptions being caught is desired.
  • migrated some uses of invokePrivileged to invokePrivilegedAndLogError
• added testing for muting behavior
  • this required adding a flag to allow Mockito access to certain parts of java.base (see commit)
  • test passes