Rename "GC Pauses" to "GC Time" in Pulsar JVM dashboards

[lhotari]

• source metric is not about GC pauses.
• the source metric is "jvm_gc_collection_seconds_sum" which is not the same as "GC pause"
• The metric is defined in Prometheus Java client
  • "Time spent in a given JVM garbage collector in seconds."
  • source code is https://github.com/prometheus/client_java/blob/e68daf23336eb5de7856df406eb1d497f51ad3be/simpleclient_hotspot/src/main/java/io/prometheus/client/hotspot/GarbageCollectorExports.java#L53
  • calls GarbageCollectorMXBean.getCollectionTime()
    • Javadoc https://docs.oracle.com/en/java/javase/11/docs/api/java.management/java/lang/management/GarbageCollectorMXBean.html#getCollectionTime() "Returns the approximate accumulated collection elapsed time in milliseconds."
    • Stackoverflow Q&A: https://stackoverflow.com/a/44686539

[dave2wave]

"GC Collection Time" is redundant. It should be "GC Time".

[lhotari]

"GC Collection Time" is redundant. It should be "GC Time".

@dave2wave thanks for pointing that out. fixed.

[lhotari]

There's also a metric for GC Pauses in Pulsar. The metric name is jvm_full_gc_pause. Here's the implementation: https://github.com/apache/pulsar/blob/52d8fe03160648f14dc196edaa87921064a1b2b2/pulsar-common/src/main/java/org/apache/pulsar/common/stats/JvmDefaultGCMetricsLogger.java#L93-L94

The implementation does raise some questions since it's not exposed directly from the source metric. Instead, the logic is attempting to calculate the duration of the most recent GC pause. There might be issues with such conversion. The source metric HotspotRuntimeMBean.getTotalSafepointTime is described in https://stackoverflow.com/a/42515743

[michaeljmarshall]

Thanks @lhotari. Is there a way to get an accurate GC pause metric?

[lhotari]

Thanks @lhotari. Is there a way to get an accurate GC pause metric?

Brokers already have a separate metric for this, jvm_full_gc_pause . https://github.com/apache/pulsar/blob/4dcb166e0bfcce7fc85fd8d59a25b881f6f9c6fa/pulsar-common/src/main/java/org/apache/pulsar/common/stats/JvmDefaultGCMetricsLogger.java#L93

It uses total safepoint time metric under the covers, https://github.com/apache/pulsar/blob/4dcb166e0bfcce7fc85fd8d59a25b881f6f9c6fa/pulsar-common/src/main/java/org/apache/pulsar/common/stats/JvmDefaultGCMetricsLogger.java#L124

It's a bit misleading to call it "full gc" since it's more than full gc. More details in https://blanco.io/blog/jvm-safepoint-pauses/