VMNodeScrape lost namespace and pod labels for cadvisor and other

[allanian]

Hello, I installed victoria-metrics-k8s-stack with helm and get issue with missed labels from cadvisor metrics (pod and namespace).

With default NodeScrape

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMNodeScrape
metadata:
  name: victoriametrics-cadvisor
  namespace: infra-victoriametrics
  resourceVersion: '321020102'
spec:
  bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  honorLabels: true
  interval: 30s
  metricRelabelConfigs:
    - action: labeldrop
      regex: (uid)
    - action: labeldrop
      regex: (id|name)
    - action: drop
      regex: >-
        (rest_client_request_duration_seconds_bucket|rest_client_request_duration_seconds_sum|rest_client_request_duration_seconds_count)
      source_labels:
        - __name__
  path: /metrics/cadvisor
  relabelConfigs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - sourceLabels:
        - __metrics_path__
      targetLabel: metrics_path
    - replacement: kubelet
      targetLabel: job
  scheme: https
  scrapeTimeout: 5s
  tlsConfig:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecureSkipVerify: true

Metrics from promql looks like:

container_memory_usage_bytes{apm="instana", beta_kubernetes_io_arch="amd64", beta_kubernetes_io_os="linux", cluster="cluster-name", container="tensorboard", feature_node_kubernetes_io_cpu_cpuid_ADX="true", feature_node_kubernetes_io_cpu_cpuid_AESNI="true", feature_node_kubernetes_io_cpu_cpuid_AVX="true", feature_node_kubernetes_io_cpu_cpuid_AVX2="true", feature_node_kubernetes_io_cpu_cpuid_AVX512BW="true", feature_node_kubernetes_io_cpu_cpuid_AVX512CD="true", feature_node_kubernetes_io_cpu_cpuid_AVX512DQ="true", feature_node_kubernetes_io_cpu_cpuid_AVX512F="true", feature_node_kubernetes_io_cpu_cpuid_AVX512VL="true", feature_node_kubernetes_io_cpu_cpuid_FMA3="true", feature_node_kubernetes_io_cpu_cpuid_HLE="true", feature_node_kubernetes_io_cpu_cpuid_IBPB="true", feature_node_kubernetes_io_cpu_cpuid_MPX="true", feature_node_kubernetes_io_cpu_cpuid_RTM="true", feature_node_kubernetes_io_cpu_cpuid_SSE4="true", feature_node_kubernetes_io_cpu_cpuid_SSE42="true", feature_node_kubernetes_io_cpu_cpuid_STIBP="true", feature_node_kubernetes_io_cpu_cpuid_VMX="true", feature_node_kubernetes_io_cpu_hardware_multithreading="true", feature_node_kubernetes_io_cpu_pstate_status="active", feature_node_kubernetes_io_cpu_pstate_turbo="true", feature_node_kubernetes_io_cpu_rdt_RDTCMT="true", feature_node_kubernetes_io_cpu_rdt_RDTL3CA="true", feature_node_kubernetes_io_cpu_rdt_RDTMBA="true"}

With empty NodeScrape

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMNodeScrape
metadata:
  name: victoriametrics-cadvisor
  namespace: infra-victoriametrics
  resourceVersion: '321020102'
spec:
  bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  honorLabels: true
  interval: 30s
  path: /metrics/cadvisor
  relabelConfigs:
    - action: labeldrop
      regex: feature_node_kubernetes_io.+
  scheme: https
  scrapeTimeout: 5s
  tlsConfig:
    caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecureSkipVerify: true

Metrics looks like:

container_memory_usage_bytes{cluster="cluster-name", container="tensorboard", id="/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59d38390_9a5f_45ab_af1e_49950f74be67.slice/cri-containerd-9b1abdf935f047d50379886ed4265c2e471e5a3f05d387bed3eb6f3504e49c70.scope", image="docker.io/tensorflow/tensorflow:2.8.0-gpu-jupyter", instance="powerbi-clickhouse", job="infra-victoriametrics/victoriametrics-cadvisor", name="9b1abdf935f047d50379886ed4265c2e471e5a3f05d387bed3eb6f3504e49c70", namespace="infra-deepvision", node="powerbi-clickhouse", pod="deepvision-b6fbd9675-ch8qm", prometheus="infra-victoriametrics/victoriametrics"}

[Haleygo]

@allanian Hello! Is this problem still relevant with you? I don't see the config in default NodeScrape will influence namespace and pod labels, can you paste the whole scrape config in vmagent?

[allanian]

yep, its solved, by update