search

F5Networks / f5-telemetry-streaming

F5 BIG-IP Telemetry Streaming
Apache License 2.0
53 stars 24 forks source link

Not getting any metrics through OpenTelemetry Consumer #266

Closed sagor999 closed 8 months ago

sagor999 commented 8 months ago

Environment

Summary

I setup OpenTelemetry_Exporter, but it doesn't export any metrics. I also have Google_Cloud_Logging consumer, and through it I see that it dumps metrics into logs, so they are there, but for some reason OpenTelemetry exporter not seeing them. I enabled debug output and I see a lot of this:

Fri, 20 Oct 2023 18:00:16 GMT - finest: [telemetry.OpenTelemetry_Exporter.f5telemetry_default::OpenTelemetry_Consumer] Event known to not contain metrics, skipping
Fri, 20 Oct 2023 18:00:16 GMT - finest: [telemetry] The data does not have anything that matches the property key tmstats
Fri, 20 Oct 2023 18:00:16 GMT - finest: [telemetry] The data does not have anything that matches the property key tmstats

In GCP Cloud Logging I see entries like this, that look like metrics to me: image

Steps To Reproduce

Steps to reproduce the behavior:

  1. Submit the following declaration: 
        class: Telemetry
    controls:
      class: Controls
      logLevel: debug
    My_System:
      class: Telemetry_System
      systemPoller:
        interval: 60
    My_Listener:
      class: Telemetry_Listener
      port: 6514
    OpenTelemetry_Consumer:
      class: Telemetry_Consumer
      type: OpenTelemetry_Exporter
      host: otel-gateway.example.com
      port: 4318
      convertBooleansToMetrics: false
      metricsPath: /v1/metrics
      exporter: protobuf
    Google_Cloud_Consumer:
      class: Telemetry_Consumer
      type: Google_Cloud_Logging
      logScope: projects
      logScopeId: ${gcp_project_id}
      logId: bigip
      serviceEmail: ${svc_acct}
      useServiceAccountToken: true

Expected Behavior

Metrics are exported via OpenTelemetry Consumer.

Actual Behavior

Metrics are not exported via OpenTelemetry.

G-gonzalezjimenez commented 8 months ago

https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/1.20/troubleshooting.html there's a specific section on how to troubleshoot this behavior. Can you go through it and see if it helps? Thank you

sagor999 commented 8 months ago

I had to run tcpdump to verify that it is indeed sending metrics. What confused me is that I was looking for metrics named f5_* and there were none (examples on the internet were all referencing metrics with f5 prefix). But bigip is sending metrics without any prefix, making it quite difficult to discover them. This is related to this issue that you guys closed unfortunately: https://github.com/F5Networks/f5-telemetry-streaming/issues/252

sagor999 commented 8 months ago

If someone else will stumble upon this: I ended up running otel collector locally on bigip box itself, with metricstransformprocessor:

    processors:
      metricstransform:
        transforms:
          - include: (.*)$$
            match_type: regexp
            action: update
            new_name: f5_$${1}

And now all f5 metrics have a proper f5_ prefix. 🎉