<Question> How to convert Cloud trace spans to metrics and explore them in Cloud Monitoring metric explorer to create charts ?

[Santosh-Kumar-Perumal]

Hi Team,

I have my java application running on GKE instrumented with Open Telemetry auto instrumentation and the traces are exported to cloud trace using google exporter-auto-0.28.0-alpha-shaded.jar I am happy with the trace and spans able to see them in cloud trace.

Now I am looking for options to extract metrics out of the spans that I see in cloud trace spans like the latency, response time etc. into useful charts in Cloud monitoring any suggestion on how to get metrics out of cloud trace spans this would really help.

Something that I can explore from metric explorer for cloud trace and then create charts for the metrics that I see in spans is what I am really looking at as I can see I don't see any metrics which GCP pulls by default from cloud trace.

When I searched, I see lot of options

1. Setup OTEL Collector and use spanmetrics connector but not really sure how the collector will be able to store the metrics.
2. Export cloud Trace spans to BigQuery and create a cloud function that can query the Bigquery tables and write them like the latency etc.. to cloud monitoring API as a custom metrics.

References https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/connector/spanmetricsconnector/README.md

@psx95 please throw some light on this.

[psx95]

Hi @Santosh-Kumar-Perumal,

1. The collector will not be storing the metrics. I haven't used spanmetrics connector before, but IIUC, this acts both as a exporter and receiver. So from the traces pipeline the connector acts a an exporter - exporting the metrics as per your config. Then in the metrics pipeline, this connector acts as a receiver for those trace metrics and they end up wherever your metric pipeline is configured to export them. The following image might clarify the process:

2. If you export trace spans to BigQuery, I don't think you would require an extra function to convert them into "Cloud Monitoring Metrics" if you only want to visualize them in a dashboard. There is a Logs Analytics Widget that lets you chart BiqQuery queries - so this could possibly fit your use-case.

[Santosh-Kumar-Perumal]

Thanks for the response @psx95 for option 1 does it needs Ops agent to collect metrics because the below documents says something on similar lines.As of now I am using Google Shaded exporter-auto jar to export traces to cloud trace.

https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent/otlp

https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent#race-features

https://cloud.google.com/trace/docs/otlp#otlp-traces

[psx95]

No, I don't think you would require to use the Ops Agent for this.

I would recommend that you refer to Choose an instrumentation approach doc to decide which approach to use. This doc lists recommendations based on the environment in which you are running your application.

Since you're using GKE, which is a managed solution, you should be able to just use the collector. The same is mentioned in the doc linked above.

Reading the page that gives an overview of managed agents in GCP, it looks like Ops Agent should be used when you're running your workloads directly on a GCE instance.

[Santosh-Kumar-Perumal]

Thanks @psx95 for the detailed explanation I understand we need to use collector for apps deployed on GKE. So, you meant to say if I export traces to OTEL collector will I be able to explore the trace metrics in cloud monitoring metric explorer?

[psx95]

Yes, if you send the traces to the OTel collector, the collector should be able to export the telemetry to Google Cloud.

The opentelemetry-collector-contrib has a googlecloudexporter, which can be used to export telemetry from collector to Google Cloud.

[psx95]

I believe the question has been answered, feel free to re-open if this is not the case.