includeMetrics doesn't seem to work as expected

[AlissonRS]

Hello!

I deployed Grafana Alloy in my cluster to send metrics/logs/traces to Grafana Cloud, though I want only certain metrics to be uploaded to reduce costs with unused metrics.

In my values.yaml I added the includeMetrics in the metrics part like this:

    metrics:
      enabled: true
      alloy:
        metricsTuning:
          useIntegrationAllowList: true
          includeMetrics: 
            - nginx_ingress_controller_request_duration_seconds_bucket
            - kube_pod_status_phase
            - node_cpu_seconds_total
            - kube_pod_container_resource_requests
            - kubelet_pod_worker_duration_seconds_bucket
            - kube_pod_container_info
            - namespace_workload_pod:kube_pod_owner:relabel
            - kube_pod_owner
            - kube_pod_info
            - kube_pod_container_resource_limits
            - container_cpu_allocation
            - container_memory_allocation_bytes
            - container_cpu_usage_seconds_total
            - container_memory_cache
            - container_memory_rss
            - container_memory_swap
            - container_memory_working_set_bytes
            - nginx_ingress_controller_request_size_sum
            - nginx_ingress_controller_requests
            - nginx_ingress_controller_response_size_sum
            - probe_http_duration_seconds
            - kube_replicaset_owner
            - kube_node_status_condition
            - kube_job_status_active
            - kube_job_status_start_time
            - rest_client_requests_total
            - kube_node_status_allocatable

Though I still see over 140k metric series, which is the same number as before, and I see metrics that are not in this list.

I raised a ticket (152801) with Grafana Support team since I have a paid plan, but it's been almost two weeks with no response, so I thought maybe I can get help here 🙏

Thanks!!!

[petewall]

Sorry to hear about your troubles with managing metrics! We'll get you all set. Here is an example that explains how the metrics tuning section works: https://github.com/grafana/k8s-monitoring-helm/tree/main/examples/custom-metrics-tuning

There are separate sections for each metric source. The first problem above is that it's all inside of the metrics.alloy section, which is about the metrics we capture from alloy itself. Also, it looks like your list of metrics to keep come from many different metric sources.

I'd recommend trying this:

metrics:
  kube-state-metrics:
    metricsTuning:
      useDefaultAllowList: false
      includeMetrics:
            - kube_pod_status_phase
            - kube_pod_container_resource_requests
            - kube_pod_container_info
            - kube_pod_owner
            - kube_pod_info
            - kube_pod_container_resource_limits
            - kube_replicaset_owner
            - kube_node_status_condition
            - kube_job_status_active
            - kube_job_status_start_time
            - kube_node_status_allocatable
  node-exporter:
    metricsTuning:
      useDefaultAllowList: false
      includeMetrics:
          - node_cpu_seconds_total
  kubelet:
    metricsTuning:
      useDefaultAllowList: false
      includeMetrics:
            - kubelet_pod_worker_duration_seconds_bucket
  cadvisor:
    metricsTuning:
      useDefaultAllowList: false
      includeMetrics:
            - container_cpu_allocation
            - container_memory_allocation_bytes
            - container_cpu_usage_seconds_total
            - container_memory_cache
            - container_memory_rss
            - container_memory_swap
            - container_memory_working_set_bytes

You don't need to add namespace_workload_pod:kube_pod_owner:relabel, because that is a recording rule that'll live inside your prometheus database. It requires the kube_pod_owner metric, which we're already passing through from kube-state-metrics.