search

prometheus / jmx_exporter

A process for exposing JMX Beans via HTTP for Prometheus consumption
Apache License 2.0
3.06k stars 1.2k forks source link

Support for JMX filter pattern for kafka metrics #908

Closed sreelu27 closed 9 months ago

sreelu27 commented 10 months ago

I am trying to create a pattern for the below Kafka metrics to be filtered: kafka_network_RequestMetrics_Count

I have tried the below pattern :

pattern : kafka.network<>Count

Even after applying this filter, I can see other metrics related to jvm and jmx.

Below is the config : 

lowercaseOutputName: false
    rules:
    - pattern : kafka.network<type=RequestMetrics, name=LocalTimeMs, request=Fetch><>Count
sreelu27 commented 10 months ago

Hi @dhoard Can we get any help on the filtering issue, please?

dhoard commented 10 months ago

@sreelu27 are you wanting any of the metrics from the RequestMetrics MBean?

sreelu27 commented 10 months ago

Hi @dhoard ,

Do you want any of the metrics from the RequestMetrics MBean ? – No

The issue here is when I apply this filter, all network related metrics are coming but along with that JMX and JVM metrics also coming that's the problem.

**jvm_threads_deadlocked_monitor 0.0

HELP jvm_threads_state Current count of threads by state

TYPE jvm_threads_state gauge

jvm_threads_state{state="NEW",} 0.0 jvm_threads_state{state="TERMINATED",} 0.0 jvm_threads_state{state="RUNNABLE",} 25.0 jvm_threads_state{state="BLOCKED",} 0.0 jvm_threads_state{state="WAITING",} 15.0 jvm_threads_state{state="TIMED_WAITING",} 36.0 jvm_threads_state{state="UNKNOWN",} 0.0**

HELP kafka_network_RequestMetrics_Count Attribute exposed for management kafka.network:name=LocalTimeMs,type=RequestMetrics,attribute=Count

TYPE kafka_network_RequestMetrics_Count untyped

kafka_network_RequestMetrics_Count{name="LocalTimeMs",request="Fetch",} 0.0

**# HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.

TYPE jmx_scrape_duration_seconds gauge

jmx_scrape_duration_seconds 0.15442887**

sreelu27 commented 10 months ago

Hi @dhoard, Can we get any help on the issue, please?

sreelu27 commented 9 months ago

Hi @dhoard, Can we get any help on the previous comment, please ? https://github.com/prometheus/jmx_exporter/issues/908#issuecomment-1916489939

sreelu27 commented 9 months ago

Hi @dhoard,

I tried to add blacklistObjectNames as below, but still I am getting the jvm metrics, jvm_threads_state{state="NEW",} 0.0 jvm_threads_state{state="TERMINATED",} 0.0 jvm_threads_state{state="RUNNABLE",} 21.0 jvm_threads_state{state="BLOCKED",} 0.0 jvm_threads_state{state="WAITING",} 17.0 jvm_threads_state{state="TIMED_WAITING",} 39.0 jvm_threads_state{state="UNKNOWN",} 0.0

    lowercaseOutputName: false
    blacklistObjectNames: ["jvm.threads*"]
    rules:
    - pattern : kafka.network<type=RequestMetrics, name=LocalTimeMs, request=Fetch><>Count
dhoard commented 9 months ago

@sreelu27 if you don't want any request metrics, then you should use...

excludeObjectNames: ["kafka.network:type=RequestMetrics,*"]

... which will filter all all MBean's with ObjectNames that match.

sreelu27 commented 9 months ago

Hi @dhoard ,

No I want to exclude the jvm metrics

jvm_memory_pool_bytes_used{pool="CodeHeap 'non-nmethods'",} 1580928.0 jvm_memory_pool_bytes_used{pool="Metaspace",} 6.7503056E7 jvm_memory_pool_bytes_used{pool="CodeHeap 'profiled nmethods'",} 1.213376E7 jvm_memory_pool_bytes_used{pool="Compressed Class Space",} 8137312.0 jvm_memory_pool_bytes_used{pool="G1 Eden Space",} 3.3554432E7 jvm_memory_pool_bytes_used{pool="G1 Old Gen",} 1.7213184E8 jvm_memory_pool_bytes_used{pool="G1 Survivor Space",} 2097152.0 jvm_memory_pool_bytes_used{pool="CodeHeap 'non-profiled nmethods'",} 2563200.0

Is there any way I can do that?

We are using jmx version 0.18.0

dhoard commented 9 months ago

@sreelu27 some questions... since this is different than what you originally requested...

I am trying to create a pattern for the below Kafka metrics to be filtered: kafka_network_RequestMetrics_Count

Are you wanting to exclude all JVM metrics? Are you running the exporter as a Java agent?

sreelu27 commented 9 months ago

Hi @dhoard , apologies for the confusion. Are you wanting to exclude all JVM metrics? yes Are you running the exporter as a Java agent? yes

dhoard commented 9 months ago

@sreelu27 you should be able to use the configuration excludeObjectNames to exclude the MBean that is providing the metrics.

sreelu27 commented 9 months ago

Hi @dhoard,

I was trying to apply the configuration excludeObjectNames using the below configmap 

kafka_metrics_config.yaml: |
    lowercaseOutputName: true
    excludeObjectNames: ["java.lang:name=G1 Young Generation,type=GarbageCollector","java.lang:name=CodeHeap 'non-profiled nmethods',type=MemoryPool","java.lang:name=CodeHeap 'profiled nmethods',type=MemoryPool"]
    rules:
    - pattern : kafka.network<type=RequestMetrics, name=LocalTimeMs, request=Fetch><>Count
curl localhost:9404 | grep CodeHeap                         

jvm_memory_pool_bytes_used{pool="CodeHeap 'non-nmethods'",} 1538688.0
261jvm_memory_pool_bytes_used{pool="CodeHeap 'profiled nmethods'",} 8927744.0
5jvm_memory_pool_bytes_used{pool="CodeHeap 'non-profiled nmethods'",} 2490880.0
  jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-nmethods'",} 2555904.0
 jvm_memory_pool_bytes_committed{pool="CodeHeap 'profiled nmethods'",} 1.1730944E7
 0     0   169k      0 --:--:-- --:--:-jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-profiled nmethods'",} 2752512.0
- jvm_memory_pool_bytes_max{pool="CodeHeap 'non-nmethods'",} 5828608.0
-jvm_memory_pool_bytes_max{pool="CodeHeap 'profiled nmethods'",} 1.22912768E8
jvm_memory_pool_bytes_max{pool="CodeHeap 'non-profiled nmethods'",} 1.22916864E8
-jvm_memory_pool_bytes_init{pool="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_pool_bytes_init{pool="CodeHeap 'profiled nmethods'",} 2555904.0
:jvm_memory_pool_bytes_init{pool="CodeHeap 'non-profiled nmethods'",} 2555904.0
--jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'",} 1.0053888E7
:--jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled nmethods'",} 2187776.0
 jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'",} 1537152.0
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'profiled nmethods'",} 1.707223262247E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-profiled nmethods'",} 1.70722326225E9
 171k
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-nmethods'",} 1.70722326225E9

I have tried, but didnt work,

https://github.com/prometheus/jmx_exporter/blob/f96796d5b242ffa9b0ec53e30091429901c8c52a/integration_test_suite/integration_tests/src/test/resources/io/prometheus/jmx/test/ExcludeObjectNamesTest/JavaAgent/exporter.yaml

kafka_metrics_config.yaml: |
    lowercaseOutputName: false
    excludeObjectNames: ["java.lang:type=ObjectName"]
    excludeObjectNameAttributes:
        "java.lang:name=Survivor Space,type=MemoryPool":
            - "ObjectName"
        "java.lang:type=OperatingSystem":
            - "ObjectName"
        "java.lang:type=Threading":
            - "ObjectName"
        "java.nio:name=direct,type=BufferPool":
            - "ObjectName"
        "java.lang:type=Compilation":
            - "ObjectName"
        "java.lang:name=Tenured Gen,type=MemoryPool":
            - "ObjectName"
        "java.lang:name=CodeCacheManager,type=MemoryManager":
            - "ObjectName"
        "java.util.logging:type=Logging":
            - "ObjectName"
        "java.lang:type=ClassLoading":
            - "ObjectName"
        "java.lang:name=Metaspace Manager,type=MemoryManager":
            - "ObjectName"
        "java.lang:name=Metaspace,type=MemoryPool":
            - "ObjectName"
        "java.lang:name=Eden Space,type=MemoryPool":
            - "ObjectName"
        "java.lang:name=CodeHeap 'profiled nmethods',type=MemoryPool":
            - "ObjectName"
        "java.lang:name=Copy,type=GarbageCollector":
            - "ObjectName"
        "java.lang:name=MarkSweepCompact,type=GarbageCollector":
            - "ObjectName"
        "java.nio:name=mapped - 'non-volatile memory',type=BufferPool":
            - "ObjectName"
        "java.lang:name=CodeHeap 'non-nmethods',type=MemoryPool":
            - "ObjectName"
        "java.lang:name=Compressed Class Space,type=MemoryPool":
            - "ObjectName"
        "java.nio:name=mapped,type=BufferPool":
            - "ObjectName"
        "java.lang:type=Memory":
            - "ObjectName"
        "java.lang:name=CodeHeap 'non-profiled nmethods',type=MemoryPool":
            - "ObjectName"
        "com.sun.management:type=HotSpotDiagnostic":
            - "ObjectName"
        "java.lang:type=Runtime":
            - "ObjectName"
            - "ClassPath"
            - "SystemProperties"
    rules:
    - pattern : kafka.network<type=RequestMetrics, name=LocalTimeMs, request=Fetch><>Count

Also tried wild card pattern :

https://github.com/prometheus/jmx_exporter/blob/f96796d5b242ffa9b0ec53e30091429901c8c52a/integration_test_suite/integration_tests/src/test/resources/io/prometheus/jmx/test/ExcludeObjectNamesTest/JavaAgent/exporter.yaml

excludeObjectNames: ["java.lang:*"]
rules:
  - pattern: ".*"

All of the above didn't work.

Is the MBean, that I am using is incorrect ? or is it that we cant exclude these default metrics (jvm & jmx) ?

Could you please help me here @dhoard ? Thanks :)

sreelu27 commented 9 months ago

HI @dhoard

Can you please help me with the query https://github.com/prometheus/jmx_exporter/issues/908#issuecomment-1929452148

Thanks, Sree

dhoard commented 9 months ago

@sreelu27 please upgrade to the latest version.

sreelu27 commented 9 months ago

Hi @dhoard ,

I am using 0.20.0 version. Uplifted from 0.18.0 to 0.20.0 for configuration excludeObjectNames

Thanks, Sree

sreelu27 commented 9 months ago

Hi @dhoard,

Could you please help me with the query https://github.com/prometheus/jmx_exporter/issues/908#issuecomment-1929452148 I am using 0.20.0 jmx exporter version

Thanks, Sree

dhoard commented 9 months ago

@sreelu27 excludeObjectNames is a global setting for an MBean. If you exclude the MBean there is no need to exclude specific attributes.

excludeObjectNames: ["java.lang:*"]

Should exclude the MBean.

sreelu27 commented 9 months ago

Hi @dhoard,

I have tried the above, but it didn't work as mentioned in my previous comment?

Screenshot at 2024-02-12 17-12-34

sreelu27 commented 9 months ago

Hi @dhoard,

Could you please help me with the above query https://github.com/prometheus/jmx_exporter/issues/908#issuecomment-1939175812

Is the MBean, that I am using is incorrect ? or is it that we cant exclude these default metrics (jvm & jmx) ?

sreelu27 commented 9 months ago

Hi @dhoard,

Could you please help me with the above query https://github.com/prometheus/jmx_exporter/issues/908#issuecomment-1939175812

Is the MBean, that I am using is incorrect ? or is it that we cant exclude these default metrics (jvm & jmx) ?

dhoard commented 9 months ago

@sreelu27 apologies, I've been extremely busy with work.

After doing some testing and research, it appears what you are trying to filter is not supported. I should have caught it earlier.

https://github.com/prometheus/jmx_exporter/issues/696

sreelu27 commented 9 months ago

Hi @dhoard,

Thank you for the confirmation.Will this be supported in the future version, the enhancement that you have linked in the previous comment : https://github.com/prometheus/jmx_exporter/issues/696

Thanks, Sree

dhoard commented 9 months ago

@sreelu27 Currently I'm are working on migration to the new Prometheus Java client_library.

I feel filtering JVM metrics is low priority since most uses/use cases I have seen want to monitor these metrics.

dhoard commented 9 months ago

Working as designed. Filtering of JVM metrics is an enhancement tracked via https://github.com/prometheus/jmx_exporter/issues/696