Closed EdwardKuenen closed 1 year ago
brusdev
commented
1 year ago @EdwardKuenen this issue should be easy to fix, maybe a check to exclude registries different from io.micrometer.prometheus.PrometheusMeterRegistry should work . Are you planning to create a pull request to fix this issue?
EdwardKuenen
commented
1 year ago Hi @brusdev , thank you for your fast response! I sure will make a PR if I can fix it.
The OpenTelemetryMeterRegistry is a subclass of io.micrometer.core.instrument.MeterRegistry.
If I check the registry for a PrometheusMeterRegistry instance it will fail and I will not get any metrics if I understand correct. The scrape is not a method of MeterRegistry, so I can not check and cast to MeterRegistry.
Do you know if I can collect metrics from Artemis with the OTEL java agent without the need for the plugin? Micrometer should be supported, but I do not get any metrics with the agent on Artemis (I disabled the line with -Dotel.metrics.exporter=none).
brusdev
commented
1 year ago I think this issue is caused by the Open Telemetry java agent that is injecting OpenTelemetryMeterRegistry registries while the artemis-prometheus-metrics-plugin only expects PrometheusMeterRegistry registries :
https://github.com/rh-messaging/artemis-prometheus-metrics-plugin/blob/v2.0.0/artemis-prometheus-metrics-plugin-servlet/src/main/java/com/redhat/amq/broker/core/server/metrics/plugins/ArtemisPrometheusMetricsPluginServlet.java#L40
public ArtemisPrometheusMetricsPluginServlet() {
Set<MeterRegistry> registries = Metrics.globalRegistry.getRegistries();
if (registries != null && registries.size() > 0) {
registry = (PrometheusMeterRegistry) registries.toArray()[0];
}
}I was thinking to add a check to exclude registries different from io.micrometer.prometheus.PrometheusMeterRegistry, i.e.
public ArtemisPrometheusMetricsPluginServlet() {
Set<MeterRegistry> registries = Metrics.globalRegistry.getRegistries();
if (registries != null && registries.size() > 0) {
for (MeterRegistry meterRegistry : registries) {
if (meterRegistry instanceof PrometheusMeterRegistry) {
registry = (PrometheusMeterRegistry) meterRegistry;
break;
}
}
}
}@brusdev I think you are correct that this solves the cast exception. I will make a PR for that and we will see if I still got any metrics with the java agent loaded.
jbertram
commented
1 year ago It's not clear to me if the problem is that there's an additional meter registry (i.e. an instance of OpenTelemetryMeterRegistry) or if the OpenTelemetry instrumentation is replacing the existing instance of PrometheusMeterRegistry with an instance of OpenTelemetryMeterRegistry so that it can intercept the metrics and report them back to OpenTelemetry. If it's the former then the fix from @brusdev should work. If it's the latter then no fix will work since the plugin must use PrometheusMeterRegistry in order to invoke scrape().
In any case, if all you want to do is gather metrics for OpenTelemetry and you don't really care about Prometheus then I believe you can just use org.apache.activemq.artemis.core.server.metrics.plugins.SimpleMetricsPlugin as the metrics plugin in broker.xml, e.g.:
<metrics>
<plugin class-name="org.apache.activemq.artemis.core.server.metrics.plugins.SimpleMetricsPlugin"/>
</metrics>If there's no plugin configured then the broker doesn't actually use Micrometer at all. This is done explicitly so that the broker doesn't have a hard dependency on Micrometer.
EdwardKuenen
commented
1 year ago I tried the SimpleMetricsPlugin, but I did not get any metrics from the OTEL java agent.
I should not have add something to the lib folder?
I did got this message in the log [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
EdwardKuenen
commented
1 year ago I made the change to the Prometheus plugin as suggested by Bruscino. Now I get the metrics by scraping /metrics with the agent loaded. The agent is not submitting metrics.
EdwardKuenen
commented
1 year ago Hi, I made your suggestion and bumped a lot of versions to a more recent version. Could you please check if this is OK?
Thank you!
brusdev
commented
1 year ago @EdwardKuenen you did more than a simple fix, nice work!!! I added some comments to the your PR.
EdwardKuenen
commented
1 year ago Thank you! I am not a very experienced developer, but I hope I can do something in return for your help!
EdwardKuenen
commented
1 year ago Thank you @brusdev and @jbertram! This solved my issue..
Describe the bug We use Apache Artemis with the artemis-prometheus-metrics-plugin. We want to use the java agent for collecting logging. After adding the java agent the metrics are not collected anymore and the /metrics endpoint could not be found anymore.
Steps to reproduce Configure Artemis to use the prometheus plugin Add the java agent to the startup command with following configuration.
What did you expect to see? The plugin keeps working and the logs are scraped.
What did you see instead? The logs are scraped, but the /metrics endpoints is not available anymore. In the log I see the stacktrace below.
What version are you using? Java agent 1.25.1
Environment OS: Ubuntu 20.04.6 LTS Runtime (if different from JDK above): OpenJdk Corretto-17.0.6.10.1
Additional context Issue at Open Telemetry Java instrumentation https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/8404
Stacktrace