cloudfoundry / cloud_controller_ng

Cloud Foundry Cloud Controller
Apache License 2.0
187 stars 356 forks source link

Missing origin in prometheus metrics #3842

Closed Samze closed 1 month ago

Samze commented 1 month ago

Issue

StatsD metrics emitted from cloud controller contain an origin: 'cc' label. This allows for easy filtering when for example looking at the firehose.

cf nozzle --no-filter | grep 'origin: "cc"'`
origin:"cc" eventType:ValueMetric timestamp:1718230908637273679 deployment:"cf-a31da169de4fe97cda42" job:"clock_global" index:"2f484dd0-d960-4251-86b3-c38f21f8a1d3" ip:"10.0.4.20" tags:<key:"product" value:"VMware Tanzu Application Service" > tags:<key:"source_id" value:"cc" > tags:<key:"system_domain" value:"...." > valueMetric:<name:"deployments.update.duration" value:9.141017 unit:"ms" >

However the prometheus metric does not contain a value from this label, e.g.

origin:"" eventType:ValueMetric timestamp:1718230843873919430 deployment:"cf-a31da169de4fe97cda42" job:"cloud_controller" index:"27c6545a-6ca1-4dc1-acb6-70e5ce5e7893" ip:"10.0.4.12" tags:<key:"instance_id" value:"27c6545a-6ca1-4dc1-acb6-70e5ce5e7893" > tags:<key:"product" value:"VMware Tanzu Application Service" > tags:<key:"source_id" value:"cloud_controller_ng" > tags:<key:"system_domain" value:"...." > valueMetric:<name:"cc_requests_outstanding_total" value:1 unit:"" >

Steps to Reproduce

Enable prometheus metrics and scraper and run

cf nozzle --no-filter | grep 'origin: "cc"'`

Expected result

You should see prometheus metrics.

Current result

You will not see any prometheus metrics.

Possible Fix