open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
2.76k stars 2.19k forks source link

[processors/filter] include drop metrics with colon in their name #31911

Closed rlanore closed 4 months ago

rlanore commented 4 months ago

Component(s)

processor/filter

What happened?

Description

Opentelemetry drop my metric if it contain colon in it's name when use processors/filter include feature

Steps to Reproduce

Deploy last opentelemetry k8s operator and add opentelemetry colector manifest

Expected Result

I must see all metric on sdout

Actual Result

I see only 'up' metric

Collector version

0.96.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
      otlp:
        protocols:
          grpc:
            endpoint: "0.0.0.0:4317"
          http:
            endpoint: "0.0.0.0:4318"
      prometheus:
  ... ...
    exporters:
      logging:
        verbosity: detailed
    extensions:
      health_check:
      pprof:
        endpoint: :1888
      zpages:
        endpoint: :55679
    processors:
      filter/limits:
        metrics:
          include:
            match_type: strict
            metric_names:
            - up
            - namespace_cpu:kube_pod_container_resource_limits:sum
            - namespace_cpu:kube_pod_container_resource_requests:sum
            - namespace_memory:kube_pod_container_resource_limits:sum
            - namespace_memory:kube_pod_container_resource_requests:sum
      batch/metrics:
        timeout: 30s
        send_batch_size: 500
    service:
      extensions: [pprof, zpages, health_check]
      pipelines:
        metrics:
          receivers: [prometheus, otlp]
          processors: [filter/limits,batch/metrics, attributes/metrics]
          exporters: [logging,otlp]
      telemetry:
        logs:
          level: debug
        metrics:
          address: 0.0.0.0:8888
          level: basic

Log output

Nothing usefull

Additional context

No response

github-actions[bot] commented 4 months ago

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

TylerHelmuth commented 4 months ago

@rlanore the filterprocessor already handles metric names with colons as it is doing a string lookup. Make sure the metrics you want to keep have the extact name namespace_cpu:kube_pod_container_resource_limits:sum. You can use a debugexporter with verbosity: detailed to view the data as the filterprocessor will see it.

rlanore commented 4 months ago

Hi @TylerHelmuth , when remove filter processors i can get value from curl prometheus server

rlanore commented 4 months ago

hummm ok I did not understood use of colon into promql. It's not the name of metrics but a recording rule computed at prometheus level. So with only 'kube_pod_container_resource_limits' in the include statement and the recording rule this works. Sorry