GoogleCloudPlatform / opentelemetry-operations-go

Apache License 2.0
127 stars 99 forks source link

feat: WithMonitoredResourceAttributes and WithMonitoredResourceDescription #850

Closed bhshkh closed 4 weeks ago

bhshkh commented 1 month ago

Context: https://github.com/googleapis/google-cloud-go/pull/10046

I'm working on exporting Bigtable client metrics to Google Cloud Monitoring. I have tried using the exporter in its current form:

CBT client side metrics In step 15 in the above diagram, the monitored resource in CreateServiceTimeSeries request has 2 fields:

  1. Type: OpenTelemetry resource with attributes gets created in step 5. The exporter uses the “cloud.platform” resource attribute value to determine the monitored resource type. Complete logic can be seen here. Problem 1: The CBT client side metrics need to be written to ‘bigtable_client_raw’ monitored resource type but the exporter does not allow writing to resource types other than the ones mentioned here
  2. Labels: OpenTelemetry resource with attributes gets created in step 5. The exporter adds these resource attributes as labels to the monitored resource. The ‘bigtable_client_raw’ monitored resource needs to written with labels: project, instance, table, cluster and zone Problem 2: Only project and instance values are known at step 5 point in time. Table is not known until data operation such as ReadRows is invoked. A Bigtable instance is a container for user's data. Instances have one or more clusters, located in different zones. When library sends requests to a Bigtable instance, those requests are handled by one of the clusters in the instance. The cluster is not known while sending the request. Cluster and zone need to be read from response headers of data operation. Thus, the existing metric exporter cannot add these labels to the monitored resource.

To resolve problem 1, solution similar to https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/pull/346 is being included in this PR.

How to include monitored resource labels?

  1. While recording the metrics, add the required labels to the metrics
  2. Exporter will use the function provided in the WithMonitoredResourceAttributes to extract the resource labels from the metric labels and add them to monitored resource. Example usage:

https://github.com/bhshkh/google-cloud-go/blob/bc1d79ae1a71a774b96ab94b875b8a29c0046501/bigtable/metric_monitoring_exporter.go#L72-L75

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 37.50000% with 10 lines in your changes are missing coverage. Please review.

Project coverage is 62.52%. Comparing base (4caace7) to head (290e9f8). Report is 19 commits behind head on main.

:exclamation: Current head 290e9f8 differs from pull request most recent head 116bd2b

Please upload reports for the commit 116bd2b to get more accurate results.

Files Patch % Lines
exporter/metric/option.go 0.00% 6 Missing :warning:
exporter/metric/metric.go 60.00% 3 Missing and 1 partial :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #850 +/- ## ========================================== + Coverage 61.03% 62.52% +1.48% ========================================== Files 56 57 +1 Lines 5903 4931 -972 ========================================== - Hits 3603 3083 -520 + Misses 2143 1688 -455 - Partials 157 160 +3 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

dashpole commented 1 month ago

cc @psx95

psx95 commented 1 month ago

cc @psx95

This looks similar to the recent request we had for the Java exporter, I will take a look into this PR.

psx95 commented 1 month ago

A similar change to the Java exporter was just approved and merged in https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/pull/346. We would likely want to follow similar logic here.

psx95 commented 4 weeks ago

Similar functionality has been added in #854.