Add support for defining internal metrics

[jvz]

Related to https://issues.apache.org/jira/browse/LOG4J2-2937 (provide counters for logger rates) and #1927, we need some sort of basic metrics API we can use internally for exporting said metrics into a metrics library. This should not be created as brand new metrics API (if we want to do that, then we'd probably start somewhere like having additional modules which implement various OpenTelemetry APIs for better reusability), but it does need to define enough of an API so that various metrics collectors can use this. I've seen some similar ideas before in various Spring libraries before Spring Framework 6 integrated Micrometer everywhere as the general metrics API, so there may be some inspiration to take.

[jack-berg]

Hi 👋 - I'm a maintainer from opentelemetry-java.

Just wanted to chime in with a few notes about the OpenTelemetry API for your consideration:

• OpenTelemetry decouples the API from the SDK. The API is a set of interfaces (with a minimal Noop default implementation) with strong backwards compatibility guarantees. The SDK is the reference implementation of the API's set of interfaces. The SDK is highly configurable and extensible. For metrics, the configurability involves the ability to provide custom MetricExporters for exporting metrics out of process (in addition to the built in exporters for prometheus and OTLP), customization of the set of exported metrics and how they are aggregated via views, and more.
• The idea behind the decoupling is that the API should do nothing (i.e. noop) until someone configures an implementation to handle the calls. Instrumentation like what is being considered for log4j can safely add a dependency on the API, relying on the strong backwards compatibility guarantees which allow users to safely resolve conflicting transitive dependencies on the API from different libraries by just updating to the latest. The SDK reference implementation of the API works great out of the box for most people, but if insufficient, a motivated user can provide their own implementation of the API.
• OpenTelemetry APIs comprise multiple telemetry signals (logs, metrics, and traces), enabling the telemetry to be correlated across signals and complementary. I.e. logs or metrics recorded in the context of an active trace have correlating metadata allowing you to stitch together the story in the backend.
• OpenTelemetry APIs / SDKs are [cross language](https://opentelemetry.io/docs/instrumentation/], allowing a familiar conceptual / configuration experience in polyglot stacks.

If you're considering adding metrics to log4j, consider using the OpenTelemetry metrics API, available at maven coordinates io.opentelemetry:opentelemetry-api. I believe it will give you the API functionality required while being highly configurable for users.

[jvz]

We've got a couple sides to metrics: defining metrics within Log4j itself, and direct support for OpenTelemetry APIs. As Log4j itself does not depend on anything besides the java.base module, we need some level of indirection to support the same thing OpenTelemetry is doing with the API/SDK split (no-op by default, require an actual metrics implementation to collect and forward them). I see that the Java OpenTelemetry API modules are designed similarly (not sure about the required Java modules there, but I assume they're similarly lightweight). Thus, I'd expect the default metrics plugin for Log4j to use OpenTelemetry since that makes it much easier to configure all those other integrations, though if third party extensions create plugins that directly use some vendor tool or whatever, that's also fine (but probably easier to do via OpenTelemetry).

From a higher level, though, we'd like to consider how to best integrate with OpenTelemetry to aid with standardizing the operational side of things, but that's a different topic.

[ppkarwasz]

This should be fixed by #2469