JMX: Tomcat sessions is always exported as 0

sky333999 commented 3 months ago

Component(s)

jmx-metrics

What happened?

Description

When using tomcat as the target system for jmx-metrics, tomcat.sessions is always emitted as 0 despite the MXBeans showing a non-zero value for the activeSessions attribute.

Steps to Reproduce

Spin up a tomcat server and install any sample web app that leverages sessions on it.
Create sessions and confirm by looking at the Tomcat manager that sessions is non-zero.

In this example, shoppingcart is a sample app and as can be seen, the sessions count is 1.
Use a tool such as jconsole or jmxterm to query MXBeans and confirm activeSessions is non-zero.

$>get -b Catalina:context=/shoppingcart,host=localhost,type=Manager activeSessions
#mbean = Catalina:context=/shoppingcart,host=localhost,type=Manager:
activeSessions = 1;

The above is the output from jmxterm to query the MXBeans. As can be seen, the value matches the expectation.

Observe the value of tomcat.sessions coming from jmx-metrics. For example, setup an OTel collector pipeline with a JMX receiver and debug exporter.

Expected Result

I expect the value of tomcat.sessions as per the definition here to also be non-zero.

Actual Result

The value of tomcat.sessions is always 0.

Component version

v1.35.0

Log output

No response

Additional context

No response

jefchien commented 3 months ago

Looked into the code a bit and I think I figured it out. It's because the MBean for the tomcat manager

https://github.com/open-telemetry/opentelemetry-java-contrib/blob/b7345dc9ac7e025ffb93409211be9c05ba71eadd/jmx-metrics/src/main/resources/target-systems/tomcat.groovy#L18-L19

and many others are being defined as a single bean

https://github.com/open-telemetry/opentelemetry-java-contrib/blob/b7345dc9ac7e025ffb93409211be9c05ba71eadd/jmx-metrics/src/main/groovy/io/opentelemetry/contrib/jmxmetrics/OtelHelper.groovy#L80-L83

when in reality, the object name includes a wildcard and after performing a query against the MBean server, Catalina:type=Manager,host=localhost,context=* resolves into multiple MBeans

Catalina:context=/shoppingcart,host=localhost,type=Manager
Catalina:context=/,host=localhost,type=Manager
etc.

What ends up happening is that the gatherer will fetch all the MBeans that match the pattern and then because it thinks it's a single MBean, it only keeps the first one.

https://github.com/open-telemetry/opentelemetry-java-contrib/blob/b7345dc9ac7e025ffb93409211be9c05ba71eadd/jmx-metrics/src/main/groovy/io/opentelemetry/contrib/jmxmetrics/MBeanHelper.groovy#L93-L99

So, the metrics don't represent an aggregated value of all the beans that match the pattern. It's just the first one, which happens to be 0.

jefchien commented 3 months ago

It looks like this is a simple case of misconfiguration. The JVM metrics are defined correctly using otel.mbeans

https://github.com/open-telemetry/opentelemetry-java-contrib/blob/b7345dc9ac7e025ffb93409211be9c05ba71eadd/jmx-metrics/src/main/resources/target-systems/jvm.groovy#L21

The Tomcat metrics just need to be updated to use otel.mbeans instead of otel.mbean.

jefchien commented 3 months ago

Not quite so simple. The gatherer needs to be updated to support aggregating metrics with the same attributes. After changing the metrics to otel.mbeans, it's calling record on each of the individual data points, which results in it dropping all but the first.

{"caller":"subprocess/subprocess.go:280","msg":"WARNING: Instrument tomcat.sessions has recorded multiple values for the same attributes: {}","kind":"receiver","name":"jmx","data_type":"metrics"}

open-telemetry / opentelemetry-java-contrib