Undesired metrics added

[ecerulm]

Describe the bug

I set amazon-cloudwatch-agent to only grab a single prometheus metric java_lang_memory_heapmemoryusage_used

              "metric_selectors": [
                "^java_lang_memory_heapmemoryusage_used"
              ]

but amazon-cloudwatch-agent is adding the following undesired metrics to the PutLogEvents

• jmx_scrape_cached_beans
• jmx_scrape_duration_seconds
• jmx_scrape_error

CloudWatch log event:

{
    "CloudWatchMetrics": [
        {
            "Namespace": "CWAgent/Prometheus",
            "Dimensions": [
                [
                    "service",
                    "service_instance",
                    "ClusterName",
                    "host",
                    "job",
                    "prom_metric_type",
                    "instance",
                    "node",
                    "application"
                ]
            ],
            "Metrics": [
                {
                    "Name": "java_lang_memory_heapmemoryusage_used",
                    "Unit": "Bytes"
                },
                {
                    "Name": "jmx_scrape_cached_beans"
                },
                {
                    "Name": "jmx_scrape_duration_seconds"
                },
                {
                    "Name": "jmx_scrape_error"
                }
            ]
        }
    ],
    "ClusterName": "tableau-dp2",
    "Timestamp": "1717502587825",
    "Version": "0",
    "application": "Tableau",
    "host": "ip-10-26-128-105",
    "instance": "127.0.0.1:12302",
    "job": "jmx",
    "node": "node1",
    "prom_metric_type": "gauge",
    "service": "vizqlservice",
    "service_instance": "2",
    "java_lang_memory_heapmemoryusage_used": 506484968,
    "jmx_scrape_cached_beans": 0,
    "jmx_scrape_duration_seconds": 0.057368237,
    "jmx_scrape_error": 0
}

the .CloudWatchMetrics.Metrics contains 4 elements where I only expected one.

I in principle don't want jmx_scrape_cached_beans, jmx_scrape_duration_seconds or jmx_scrape_error as a cloudwatch metric at all (specially I don't want to pay for those) .

I believe it's a bug to be sending those since I explicitly set only java_lang_memory_heapmemoryusage_used unless I misunderstood the configuration , if so , is there any way to prevent those jmx_xxx metrics to be included?

Steps to reproduce If possible, provide a recipe for reproducing the error.

What did you expect to see? A clear and concise description of what you expected to see.

What did you see instead? A clear and concise description of what you saw instead.

What version did you use? Version: Version: CWAgent/1.300039.0b612 (go1.22.2; linux; amd64)

What config did you use? config.json

{
  "agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "root",
    "debug": true
  },
  "metrics": {
    "aggregation_dimensions": [
      [
        "InstanceId"
      ]
    ],
    "append_dimensions": {
      "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
      "ImageId": "${aws:ImageId}",
      "InstanceId": "${aws:InstanceId}",
      "InstanceType": "${aws:InstanceType}"
    },
    "metrics_collected": {
      "collectd": {
        "metrics_aggregation_interval": 60
      },
      "cpu": {
        "measurement": [
          "cpu_usage_idle",
          "cpu_usage_iowait",
          "cpu_usage_user",
          "cpu_usage_system"
        ],
        "metrics_collection_interval": 60,
        "totalcpu": true
      },
      "disk": {
        "measurement": [
          "used_percent",
          "inodes_free"
        ],
        "metrics_collection_interval": 60,
        "resources": [
          "/"
        ]
      },
      "diskio": {
        "measurement": [
          "io_time",
          "write_bytes",
          "read_bytes",
          "writes",
          "reads"
        ],
        "metrics_collection_interval": 60,
        "resources": [
          "*"
        ]
      },
      "mem": {
        "measurement": [
          "mem_used_percent"
        ],
        "metrics_collection_interval": 60
      },
      "netstat": {
        "measurement": [
          "tcp_established",
          "tcp_time_wait"
        ],
        "metrics_collection_interval": 60
      },
      "statsd": {
        "metrics_aggregation_interval": 60,
        "metrics_collection_interval": 10,
        "service_address": ":8125"
      },
      "swap": {
        "measurement": [
          "swap_used_percent"
        ],
        "metrics_collection_interval": 60
      }
    }
  },
  "logs": {
    "metrics_collected": {
      "prometheus": {
        "cluster_name": "tableau-dp2",
        "log_group_name": "tableau-dp2",
        "prometheus_config_path": "/opt/aws/amazon-cloudwatch-agent/etc/prometheus.yaml",
        "emf_processor": {
          "metric_declaration_dedup": true,
          "metric_namespace": "CWAgent/Prometheus",
          "metric_unit": {
            "java_lang_memory_heapmemoryusage_used": "Bytes"
          },
          "metric_declaration": [
            {
              "source_labels": ["node"],
              "label_matcher": "*",
              "dimensions": [
                [
                  "ClusterName",
                  "node",
                  "application",
                  "service",
                  "service_instance"
                ]
              ],
              "metric_selectors": [
                "^java_lang_memory_heapmemoryusage_used"
              ]
            }
          ]
        }
      }
    },
    "force_flush_interval": 5
  }
} 

prometheus.yaml

global:
  scrape_interval: 1m
  scrape_timeout: 10s
scrape_configs:
  - job_name: jmx
    sample_limit: 10000
    file_sd_configs:
      - files: ["/opt/aws/amazon-cloudwatch-agent/etc/prometheus_sd_jmx.yaml"]

prometheus_sd_jmx.yaml

- targets:
  - 127.0.0.1:12300
  labels:
    application: Tableau
    service: vizqlservice
    service_instance: "0"
    node: node1
- targets:
  - 127.0.0.1:12301
  labels:
    application: Tableau
    service: vizqlservice
    service_instance: "1"
    node: node1
- targets:
  - 127.0.0.1:12302
  labels:
    application: Tableau
    service: vizqlservice
    service_instance: "2"
    node: node1
- targets:
  - 127.0.0.1:12303
  labels:
    application: Tableau
    service: vizqlservice
    service_instance: "3"
    node: node1

Environment OS: (e.g., "Ubuntu 20.04")

Additional context Add any other context about the problem here.

[ecerulm]

curl -s localhost:12300/metrics
# HELP catalina_globalrequestprocessor_bytesreceived Amount of data received, in bytes Catalina:name="http-nio-8033",type=GlobalRequestProcessor,attribute=bytesReceived
# TYPE catalina_globalrequestprocessor_bytesreceived untyped
catalina_globalrequestprocessor_bytesreceived{name="\"http-nio-8033\""} 1280106.0
# HELP catalina_globalrequestprocessor_bytessent Amount of data sent, in bytes Catalina:name="http-nio-8033",type=GlobalRequestProcessor,attribute=bytesSent
# TYPE catalina_globalrequestprocessor_bytessent untyped
catalina_globalrequestprocessor_bytessent{name="\"http-nio-8033\""} 7.28018507E8
# HELP catalina_globalrequestprocessor_errorcount Number of errors Catalina:name="http-nio-8033",type=GlobalRequestProcessor,attribute=errorCount
# TYPE catalina_globalrequestprocessor_errorcount untyped
catalina_globalrequestprocessor_errorcount{name="\"http-nio-8033\""} 58.0
# HELP catalina_globalrequestprocessor_maxtime Maximum time to process a request Catalina:name="http-nio-8033",type=GlobalRequestProcessor,attribute=maxTime
# TYPE catalina_globalrequestprocessor_maxtime untyped
catalina_globalrequestprocessor_maxtime{name="\"http-nio-8033\""} 31332.0
# HELP catalina_globalrequestprocessor_processingtime Total time to process the requests Catalina:name="http-nio-8033",type=GlobalRequestProcessor,attribute=processingTime
# TYPE catalina_globalrequestprocessor_processingtime untyped
catalina_globalrequestprocessor_processingtime{name="\"http-nio-8033\""} 2083689.0
# HELP catalina_globalrequestprocessor_requestcount Number of requests processed Catalina:name="http-nio-8033",type=GlobalRequestProcessor,attribute=requestCount
# TYPE catalina_globalrequestprocessor_requestcount untyped
catalina_globalrequestprocessor_requestcount{name="\"http-nio-8033\""} 5729.0
# HELP catalina_manager_activesessions Number of active sessions at this moment Catalina:name=null,type=Manager,attribute=activeSessions
# TYPE catalina_manager_activesessions untyped
catalina_manager_activesessions{context="/vizql",host="localhost"} 0.0
# HELP catalina_manager_duplicates Number of duplicated session ids generated Catalina:name=null,type=Manager,attribute=duplicates
# TYPE catalina_manager_duplicates untyped
catalina_manager_duplicates{context="/vizql",host="localhost"} 0.0
# HELP catalina_manager_expiredsessions Number of sessions that expired ( doesn't include explicit invalidations ) Catalina:name=null,type=Manager,attribute=expiredSessions
# TYPE catalina_manager_expiredsessions untyped
catalina_manager_expiredsessions{context="/vizql",host="localhost"} 203.0
# HELP catalina_manager_maxactive Maximum number of active sessions so far Catalina:name=null,type=Manager,attribute=maxActive
# TYPE catalina_manager_maxactive untyped
catalina_manager_maxactive{context="/vizql",host="localhost"} 59.0
# HELP catalina_manager_maxactivesessions The maximum number of active Sessions allowed, or -1                        for no limit Catalina:name=null,type=Manager,attribute=maxActiveSessions
# TYPE catalina_manager_maxactivesessions untyped
catalina_manager_maxactivesessions{context="/vizql",host="localhost"} -1.0
# HELP catalina_manager_persistauthentication Indicates whether sessions shall persist authentication information when being persisted (e.g. across application restarts). Catalina:name=null,type=Manager,attribute=persistAuthentication
# TYPE catalina_manager_persistauthentication untyped
catalina_manager_persistauthentication{context="/vizql",host="localhost"} 0.0
# HELP catalina_manager_processexpiresfrequency The frequency of the manager checks (expiration and passivation) Catalina:name=null,type=Manager,attribute=processExpiresFrequency
# TYPE catalina_manager_processexpiresfrequency untyped
catalina_manager_processexpiresfrequency{context="/vizql",host="localhost"} 6.0
# HELP catalina_manager_processingtime Time spent doing housekeeping and expiration Catalina:name=null,type=Manager,attribute=processingTime
# TYPE catalina_manager_processingtime untyped
catalina_manager_processingtime{context="/vizql",host="localhost"} 21.0
# HELP catalina_manager_rejectedsessions Number of sessions we rejected due to maxActive being reached Catalina:name=null,type=Manager,attribute=rejectedSessions
# TYPE catalina_manager_rejectedsessions untyped
catalina_manager_rejectedsessions{context="/vizql",host="localhost"} 0.0
# HELP catalina_manager_sessionaveragealivetime Average time an expired session had been alive Catalina:name=null,type=Manager,attribute=sessionAverageAliveTime
# TYPE catalina_manager_sessionaveragealivetime untyped
catalina_manager_sessionaveragealivetime{context="/vizql",host="localhost"} 18.0
# HELP catalina_manager_sessioncounter Total number of sessions created by this manager Catalina:name=null,type=Manager,attribute=sessionCounter
# TYPE catalina_manager_sessioncounter untyped
catalina_manager_sessioncounter{context="/vizql",host="localhost"} 202.0
# HELP catalina_manager_sessioncreaterate Session creation rate in sessions per minute Catalina:name=null,type=Manager,attribute=sessionCreateRate
# TYPE catalina_manager_sessioncreaterate untyped
catalina_manager_sessioncreaterate{context="/vizql",host="localhost"} 0.0
# HELP catalina_manager_sessionexpirerate Session expiration rate in sessions per minute Catalina:name=null,type=Manager,attribute=sessionExpireRate
# TYPE catalina_manager_sessionexpirerate untyped
catalina_manager_sessionexpirerate{context="/vizql",host="localhost"} 0.0
# HELP catalina_manager_sessionmaxalivetime Longest time an expired session had been alive Catalina:name=null,type=Manager,attribute=sessionMaxAliveTime
# TYPE catalina_manager_sessionmaxalivetime untyped
catalina_manager_sessionmaxalivetime{context="/vizql",host="localhost"} 2213.0
# HELP catalina_manager_warnonsessionattributefilterfailure Should a WARN level log message be generated if a session attribute fails to match sessionAttributeNameFilter or sessionAttributeClassNameFilter? Catalina:name=null,type=Manager,attribute=warnOnSessionAttributeFilterFailure
# TYPE catalina_manager_warnonsessionattributefilterfailure untyped
catalina_manager_warnonsessionattributefilterfailure{context="/vizql",host="localhost"} 0.0
# HELP java_lang_classloading_loadedclasscount java.lang:name=null,type=ClassLoading,attribute=LoadedClassCount
# TYPE java_lang_classloading_loadedclasscount untyped
java_lang_classloading_loadedclasscount 40878.0
# HELP java_lang_classloading_totalloadedclasscount java.lang:name=null,type=ClassLoading,attribute=TotalLoadedClassCount
# TYPE java_lang_classloading_totalloadedclasscount untyped
java_lang_classloading_totalloadedclasscount 43298.0
# HELP java_lang_classloading_unloadedclasscount java.lang:name=null,type=ClassLoading,attribute=UnloadedClassCount
# TYPE java_lang_classloading_unloadedclasscount untyped
java_lang_classloading_unloadedclasscount 2420.0
# HELP java_lang_classloading_verbose java.lang:name=null,type=ClassLoading,attribute=Verbose
# TYPE java_lang_classloading_verbose untyped
java_lang_classloading_verbose 0.0
# HELP java_lang_g1_old_generation_collectioncount java.lang:name=G1 Old Generation,type=GarbageCollector,attribute=CollectionCount
# TYPE java_lang_g1_old_generation_collectioncount untyped
java_lang_g1_old_generation_collectioncount{type="GarbageCollector"} 0.0
# HELP java_lang_g1_old_generation_collectiontime java.lang:name=G1 Old Generation,type=GarbageCollector,attribute=CollectionTime
# TYPE java_lang_g1_old_generation_collectiontime untyped
java_lang_g1_old_generation_collectiontime{type="GarbageCollector"} 0.0
# HELP java_lang_g1_old_generation_valid java.lang:name=G1 Old Generation,type=GarbageCollector,attribute=Valid
# TYPE java_lang_g1_old_generation_valid untyped
java_lang_g1_old_generation_valid{type="GarbageCollector"} 1.0
# HELP java_lang_g1_young_generation_collectioncount java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=CollectionCount
# TYPE java_lang_g1_young_generation_collectioncount untyped
java_lang_g1_young_generation_collectioncount{type="GarbageCollector"} 139.0
# HELP java_lang_g1_young_generation_collectiontime java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=CollectionTime
# TYPE java_lang_g1_young_generation_collectiontime untyped
java_lang_g1_young_generation_collectiontime{type="GarbageCollector"} 4233.0
# HELP java_lang_g1_young_generation_lastgcinfo_duration CompositeType for GC info for G1 Young Generation java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=duration
# TYPE java_lang_g1_young_generation_lastgcinfo_duration untyped
java_lang_g1_young_generation_lastgcinfo_duration{type="GarbageCollector"} 5.0
# HELP java_lang_g1_young_generation_lastgcinfo_endtime CompositeType for GC info for G1 Young Generation java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=endTime
# TYPE java_lang_g1_young_generation_lastgcinfo_endtime untyped
java_lang_g1_young_generation_lastgcinfo_endtime{type="GarbageCollector"} 8.6729938E7
# HELP java_lang_g1_young_generation_lastgcinfo_gcthreadcount CompositeType for GC info for G1 Young Generation java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=GcThreadCount
# TYPE java_lang_g1_young_generation_lastgcinfo_gcthreadcount untyped
java_lang_g1_young_generation_lastgcinfo_gcthreadcount{type="GarbageCollector"} 5.0
# HELP java_lang_g1_young_generation_lastgcinfo_id CompositeType for GC info for G1 Young Generation java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=id
# TYPE java_lang_g1_young_generation_lastgcinfo_id untyped
java_lang_g1_young_generation_lastgcinfo_id{type="GarbageCollector"} 139.0
# HELP java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed java.lang.management.MemoryUsage java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=committed
# TYPE java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed untyped
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed{key="CodeHeap 'non-nmethods'",type="GarbageCollector"} 4849664.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed{key="CodeHeap 'non-profiled nmethods'",type="GarbageCollector"} 4.161536E7
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed{key="CodeHeap 'profiled nmethods'",type="GarbageCollector"} 8.0412672E7
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed{key="Compressed Class Space",type="GarbageCollector"} 3.2243712E7
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed{key="G1 Eden Space",type="GarbageCollector"} 7.56023296E8
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed{key="G1 Old Gen",type="GarbageCollector"} 6.6060288E8
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed{key="G1 Survivor Space",type="GarbageCollector"} 5242880.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_committed{key="Metaspace",type="GarbageCollector"} 2.42782208E8
# HELP java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init java.lang.management.MemoryUsage java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=init
# TYPE java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init untyped
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init{key="CodeHeap 'non-nmethods'",type="GarbageCollector"} 2555904.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init{key="CodeHeap 'non-profiled nmethods'",type="GarbageCollector"} 2555904.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init{key="CodeHeap 'profiled nmethods'",type="GarbageCollector"} 2555904.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init{key="Compressed Class Space",type="GarbageCollector"} 0.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init{key="G1 Eden Space",type="GarbageCollector"} 2.7262976E7
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init{key="G1 Old Gen",type="GarbageCollector"} 3.9845888E7
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init{key="G1 Survivor Space",type="GarbageCollector"} 0.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_init{key="Metaspace",type="GarbageCollector"} 0.0
# HELP java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max java.lang.management.MemoryUsage java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=max
# TYPE java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max untyped
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max{key="CodeHeap 'non-nmethods'",type="GarbageCollector"} 8183808.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max{key="CodeHeap 'non-profiled nmethods'",type="GarbageCollector"} 1.21737216E8
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max{key="CodeHeap 'profiled nmethods'",type="GarbageCollector"} 1.21737216E8
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max{key="Compressed Class Space",type="GarbageCollector"} 1.073741824E9
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max{key="G1 Eden Space",type="GarbageCollector"} -1.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max{key="G1 Old Gen",type="GarbageCollector"} 8.355053568E9
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max{key="G1 Survivor Space",type="GarbageCollector"} -1.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_max{key="Metaspace",type="GarbageCollector"} -1.0
# HELP java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used java.lang.management.MemoryUsage java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=used
# TYPE java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used untyped
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used{key="CodeHeap 'non-nmethods'",type="GarbageCollector"} 1952128.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used{key="CodeHeap 'non-profiled nmethods'",type="GarbageCollector"} 4.1590016E7
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used{key="CodeHeap 'profiled nmethods'",type="GarbageCollector"} 7.8974208E7
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used{key="Compressed Class Space",type="GarbageCollector"} 2.8333752E7
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used{key="G1 Eden Space",type="GarbageCollector"} 0.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used{key="G1 Old Gen",type="GarbageCollector"} 5.0559672E8
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used{key="G1 Survivor Space",type="GarbageCollector"} 5242880.0
java_lang_g1_young_generation_lastgcinfo_memoryusageaftergc_used{key="Metaspace",type="GarbageCollector"} 2.30296208E8
# HELP java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed java.lang.management.MemoryUsage java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=committed
# TYPE java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed untyped
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed{key="CodeHeap 'non-nmethods'",type="GarbageCollector"} 4849664.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed{key="CodeHeap 'non-profiled nmethods'",type="GarbageCollector"} 4.161536E7
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed{key="CodeHeap 'profiled nmethods'",type="GarbageCollector"} 8.0412672E7
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed{key="Compressed Class Space",type="GarbageCollector"} 3.2243712E7
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed{key="G1 Eden Space",type="GarbageCollector"} 7.66509056E8
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed{key="G1 Old Gen",type="GarbageCollector"} 6.53262848E8
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed{key="G1 Survivor Space",type="GarbageCollector"} 2097152.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_committed{key="Metaspace",type="GarbageCollector"} 2.42782208E8
# HELP java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init java.lang.management.MemoryUsage java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=init
# TYPE java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init untyped
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init{key="CodeHeap 'non-nmethods'",type="GarbageCollector"} 2555904.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init{key="CodeHeap 'non-profiled nmethods'",type="GarbageCollector"} 2555904.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init{key="CodeHeap 'profiled nmethods'",type="GarbageCollector"} 2555904.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init{key="Compressed Class Space",type="GarbageCollector"} 0.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init{key="G1 Eden Space",type="GarbageCollector"} 2.7262976E7
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init{key="G1 Old Gen",type="GarbageCollector"} 3.9845888E7
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init{key="G1 Survivor Space",type="GarbageCollector"} 0.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_init{key="Metaspace",type="GarbageCollector"} 0.0
# HELP java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max java.lang.management.MemoryUsage java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=max
# TYPE java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max untyped
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max{key="CodeHeap 'non-nmethods'",type="GarbageCollector"} 8183808.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max{key="CodeHeap 'non-profiled nmethods'",type="GarbageCollector"} 1.21737216E8
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max{key="CodeHeap 'profiled nmethods'",type="GarbageCollector"} 1.21737216E8
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max{key="Compressed Class Space",type="GarbageCollector"} 1.073741824E9
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max{key="G1 Eden Space",type="GarbageCollector"} -1.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max{key="G1 Old Gen",type="GarbageCollector"} 8.355053568E9
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max{key="G1 Survivor Space",type="GarbageCollector"} -1.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_max{key="Metaspace",type="GarbageCollector"} -1.0
# HELP java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used java.lang.management.MemoryUsage java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=used
# TYPE java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used untyped
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used{key="CodeHeap 'non-nmethods'",type="GarbageCollector"} 1952128.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used{key="CodeHeap 'non-profiled nmethods'",type="GarbageCollector"} 4.1590016E7
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used{key="CodeHeap 'profiled nmethods'",type="GarbageCollector"} 7.8974208E7
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used{key="Compressed Class Space",type="GarbageCollector"} 2.8333752E7
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used{key="G1 Eden Space",type="GarbageCollector"} 2.81018368E8
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used{key="G1 Old Gen",type="GarbageCollector"} 5.055452E8
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used{key="G1 Survivor Space",type="GarbageCollector"} 2097152.0
java_lang_g1_young_generation_lastgcinfo_memoryusagebeforegc_used{key="Metaspace",type="GarbageCollector"} 2.30296208E8
# HELP java_lang_g1_young_generation_lastgcinfo_starttime CompositeType for GC info for G1 Young Generation java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=startTime
# TYPE java_lang_g1_young_generation_lastgcinfo_starttime untyped
java_lang_g1_young_generation_lastgcinfo_starttime{type="GarbageCollector"} 8.6729933E7
# HELP java_lang_g1_young_generation_valid java.lang:name=G1 Young Generation,type=GarbageCollector,attribute=Valid
# TYPE java_lang_g1_young_generation_valid untyped
java_lang_g1_young_generation_valid{type="GarbageCollector"} 1.0
# HELP java_lang_memory_heapmemoryusage_committed java.lang.management.MemoryUsage java.lang:name=null,type=Memory,attribute=committed
# TYPE java_lang_memory_heapmemoryusage_committed untyped
java_lang_memory_heapmemoryusage_committed 1.421869056E9
# HELP java_lang_memory_heapmemoryusage_init java.lang.management.MemoryUsage java.lang:name=null,type=Memory,attribute=init
# TYPE java_lang_memory_heapmemoryusage_init untyped
java_lang_memory_heapmemoryusage_init 6.7108864E7
# HELP java_lang_memory_heapmemoryusage_max java.lang.management.MemoryUsage java.lang:name=null,type=Memory,attribute=max
# TYPE java_lang_memory_heapmemoryusage_max untyped
java_lang_memory_heapmemoryusage_max 8.355053568E9
# HELP java_lang_memory_heapmemoryusage_used java.lang.management.MemoryUsage java.lang:name=null,type=Memory,attribute=used
# TYPE java_lang_memory_heapmemoryusage_used untyped
java_lang_memory_heapmemoryusage_used 9.10347056E8
# HELP java_lang_threading_currentthreadallocatedbytes java.lang:name=null,type=Threading,attribute=CurrentThreadAllocatedBytes
# TYPE java_lang_threading_currentthreadallocatedbytes untyped
java_lang_threading_currentthreadallocatedbytes 8136312.0
# HELP java_lang_threading_currentthreadcputime java.lang:name=null,type=Threading,attribute=CurrentThreadCpuTime
# TYPE java_lang_threading_currentthreadcputime untyped
java_lang_threading_currentthreadcputime 7.6256619E7
# HELP java_lang_threading_currentthreadcputimesupported java.lang:name=null,type=Threading,attribute=CurrentThreadCpuTimeSupported
# TYPE java_lang_threading_currentthreadcputimesupported untyped
java_lang_threading_currentthreadcputimesupported 1.0
# HELP java_lang_threading_currentthreadusertime java.lang:name=null,type=Threading,attribute=CurrentThreadUserTime
# TYPE java_lang_threading_currentthreadusertime untyped
java_lang_threading_currentthreadusertime 6.0E7
# HELP java_lang_threading_daemonthreadcount java.lang:name=null,type=Threading,attribute=DaemonThreadCount
# TYPE java_lang_threading_daemonthreadcount untyped
java_lang_threading_daemonthreadcount 399.0
# HELP java_lang_threading_objectmonitorusagesupported java.lang:name=null,type=Threading,attribute=ObjectMonitorUsageSupported
# TYPE java_lang_threading_objectmonitorusagesupported untyped
java_lang_threading_objectmonitorusagesupported 1.0
# HELP java_lang_threading_peakthreadcount java.lang:name=null,type=Threading,attribute=PeakThreadCount
# TYPE java_lang_threading_peakthreadcount untyped
java_lang_threading_peakthreadcount 679.0
# HELP java_lang_threading_synchronizerusagesupported java.lang:name=null,type=Threading,attribute=SynchronizerUsageSupported
# TYPE java_lang_threading_synchronizerusagesupported untyped
java_lang_threading_synchronizerusagesupported 1.0
# HELP java_lang_threading_threadallocatedmemoryenabled java.lang:name=null,type=Threading,attribute=ThreadAllocatedMemoryEnabled
# TYPE java_lang_threading_threadallocatedmemoryenabled untyped
java_lang_threading_threadallocatedmemoryenabled 1.0
# HELP java_lang_threading_threadallocatedmemorysupported java.lang:name=null,type=Threading,attribute=ThreadAllocatedMemorySupported
# TYPE java_lang_threading_threadallocatedmemorysupported untyped
java_lang_threading_threadallocatedmemorysupported 1.0
# HELP java_lang_threading_threadcontentionmonitoringenabled java.lang:name=null,type=Threading,attribute=ThreadContentionMonitoringEnabled
# TYPE java_lang_threading_threadcontentionmonitoringenabled untyped
java_lang_threading_threadcontentionmonitoringenabled 0.0
# HELP java_lang_threading_threadcontentionmonitoringsupported java.lang:name=null,type=Threading,attribute=ThreadContentionMonitoringSupported
# TYPE java_lang_threading_threadcontentionmonitoringsupported untyped
java_lang_threading_threadcontentionmonitoringsupported 1.0
# HELP java_lang_threading_threadcount java.lang:name=null,type=Threading,attribute=ThreadCount
# TYPE java_lang_threading_threadcount untyped
java_lang_threading_threadcount 670.0
# HELP java_lang_threading_threadcputimeenabled java.lang:name=null,type=Threading,attribute=ThreadCpuTimeEnabled
# TYPE java_lang_threading_threadcputimeenabled untyped
java_lang_threading_threadcputimeenabled 1.0
# HELP java_lang_threading_threadcputimesupported java.lang:name=null,type=Threading,attribute=ThreadCpuTimeSupported
# TYPE java_lang_threading_threadcputimesupported untyped
java_lang_threading_threadcputimesupported 1.0
# HELP java_lang_threading_totalstartedthreadcount java.lang:name=null,type=Threading,attribute=TotalStartedThreadCount
# TYPE java_lang_threading_totalstartedthreadcount untyped
java_lang_threading_totalstartedthreadcount 5614.0
# HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_total counter
jmx_config_reload_failure_total 0.0
# HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_total counter
jmx_config_reload_success_total 1.0
# HELP jmx_exporter_build_info JMX Exporter build information
# TYPE jmx_exporter_build_info gauge
jmx_exporter_build_info{name="jmx_prometheus_httpserver",version="1.0.1"} 1
# HELP jmx_scrape_cached_beans Number of beans with their matching rule cached
# TYPE jmx_scrape_cached_beans gauge
jmx_scrape_cached_beans 0.0
# HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.
# TYPE jmx_scrape_duration_seconds gauge
jmx_scrape_duration_seconds 0.008187466
# HELP jmx_scrape_error Non-zero if this scrape failed.
# TYPE jmx_scrape_error gauge
jmx_scrape_error 0.0

as you can see there is a lot of jmx_* metrics but for some reason amazon-cloudwatch-agent is grabbing only

• jmx_scrape_cached_beans
• jmx_scrape_duration_seconds
• jmx_scrape_error

although I explicitly said I don't want any of those jmx_ since I used "^java_lang_memory_heapmemoryusage_used" as metric_selectors

I principle I expected only to see java_lang_memory_heapmemoryusage_used as the only metric exported to CW.