search

micrometer-metrics / micrometer

An application observability facade for the most popular observability tools. Think SLF4J, but for observability.
https://micrometer.io
Apache License 2.0
4.49k stars 993 forks source link

Missing memory metrics area with Oracle Graalvm native mode and GC G1 #5697

Closed raafael closed 1 week ago

raafael commented 1 week ago

Describe the bug A clear and concise description of what the bug is.

I don't know if it's a bug but if I build a Quarkus application with micrometer extension, Oracle graalvm in native mode and using GC G1 some memory metrics are missing areas:

# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{application="my-app-g1",area="nonheap",id="runtime code cache (code and data)",} 0.0
jvm_memory_used_bytes{application="my-app-g1",area="nonheap",id="runtime code cache (native metadata)",} 0.0

The same application compile with Oracle graalvm + native but with Serial GC I get more areas:

# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{application="my-app-sgc",area="heap",id="old generation space",} 3145728.0
jvm_memory_used_bytes{application="my-app-sgc",area="heap",id="eden space",} 524288.0
jvm_memory_used_bytes{application="my-app-sgc",area="nonheap",id="runtime code cache (code and data)",} 0.0
jvm_memory_used_bytes{application="my-app-sgc",area="nonheap",id="runtime code cache (native metadata)",} 0.0
jvm_memory_used_bytes{application="my-app-sgc",area="heap",id="survivor space",} 0.0

And if I run in normal mode with G1 I get more areas too:

# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{application="my-app",area="nonheap",id="Compressed Class Space",} 6299704.0
jvm_memory_used_bytes{application="my-app",area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 1321600.0
jvm_memory_used_bytes{application="my-app",area="nonheap",id="CodeHeap 'profiled nmethods'",} 8398464.0
jvm_memory_used_bytes{application="my-app",area="heap",id="G1 Survivor Space",} 3485328.0
jvm_memory_used_bytes{application="my-app",area="nonheap",id="CodeHeap 'non-nmethods'",} 2291968.0
jvm_memory_used_bytes{application="my-app",area="heap",id="G1 Eden Space",} 1.6777216E7
jvm_memory_used_bytes{application="my-app",area="heap",id="G1 Old Gen",} 1.4199376E7
jvm_memory_used_bytes{application="my-app",area="nonheap",id="Metaspace",} 5.4121104E7

Environment native compilation

Environment local no native compilation

To Reproduce How to reproduce the bug:

Expected behavior I expected that I will have the same information with GC G1 in both compile modes: normal and native

Additional context Add any other context about the problem here, e.g. related issues.

jonatan-ivanov commented 1 week ago

Could you please try GraalVM 23?

Most probably this information is not available on GraalVM right now (JMX support is incomplete on GraalVM). You can verify this via either debugging JvmMemoryMetrics or printing out contents of MemoryPoolMXBean manually.

jonatan-ivanov commented 1 week ago

Oh it seems it is not supported yet: https://github.com/oracle/graal/issues/8367 Please follow up on this issue. Once it is implemented in GraalVM, metrics should show up automatically.

raafael commented 1 week ago

@jonatan-ivanov thanks for your reply I will follow the issue, something that I try to do was to build and run the app compiled in native mode with jmx support and in my local environment , I could see the memory information with visualvm but I couldn't find how to expose to prometheus. I found the Prometheus JMX Exporter but graalvm is not ready to support agents yet . Did you do something like that? Export with graalvm native to prometheus jmx metrics with no agent?

jonatan-ivanov commented 1 week ago

I could see the memory information with visualvm

That's weird, are you sure it was a native-image app? If so and you can figure out how to get that information programmatically, e.g.: print to stdout (JMX lets you query everything), I can help you exposing it (we just need to create a Gauge).

raafael commented 1 week ago

@jonatan-ivanov Yes I am sure, I even did a docker pull from the same image deployed in k8s environment to test local with jmx. Wondering if I can use the Prometheus JMX Exporter and add somehow in my quarkus app to scrape those information from jmx to prometheus... Or if it exists another way I am appreciate your help to point some documentation from where I could start.

jonatan-ivanov commented 6 days ago

I'm not sure, since the exporter is a java agent and I'm not sure if it can be used as part of the app. But even so, I don't think I would recommend it since you need to write brittle regexp to query values from JMX. If you can come up with such a regexp, you can also query those values in your source code and bind them to a Gauge (that's what I was referring to above).

But let's say you can use the JMX agent, if you tell me what you would write into the path/pattern of its config file to get the values you want, I can help you creating a MeterBinder that does the same in Java.