CloudTrace : understanding how things are connected

[glb-cblin]

I have a PubSub pull subscriber running on Cloud Run

I'd like to create a new trace for each message AND see the trace "metrics" like for a HTTP request, especially the duration of the trace

At the moment, I'm using this pieece of code (combination of various findings in .NET and GCP docs + multiple tries + previous issues I created on this github) :

using var activity = new Activity($"Processing Message {message.MessageId}").Start();
try
{
    logger.LogInformation("Type {MessageType} MessageId: {MessageId}", "PubSubMessage", message.MessageId);

    // https://cloud.google.com/pubsub/docs/publish-receive-messages-client-library#c_1
    var data = System.Text.Encoding.UTF8.GetString(message.Data.ToArray());

    logger.LogInformation("Message content: Data: {Data}, Attributes: {Attributes}, MessageId: {MessageId}", data, message.Attributes, message.MessageId);
    await ProcessMessage(data!);

    activity.SetStatus(ActivityStatusCode.Ok);
    return SubscriberClient.Reply.Ack;
}
catch (Exception e)
{
    activity.SetStatus(ActivityStatusCode.Error, e.Message);
    logger.LogError(e.Message);
    return SubscriberClient.Reply.Nack;
}
finally
{
    activity.Stop();
}

The logs are correctly correlated but the trace is not working : I do not have the duration metrics and the icon is not working

I suppose I need to manually integrate with Cloud Trace API to start a Trace

So I looked here : https://cloud.google.com/dotnet/docs/reference/Google.Cloud.Trace.V2/latest and https://cloud.google.com/dotnet/docs/reference/Google.Cloud.Trace.V2/latest/Google.Cloud.Trace.V2.TraceServiceClient#Google_Cloud_Trace_V2_TraceServiceClient_CreateSpanAsync_Google_Cloud_Trace_V2_Span_System_Threading_CancellationToken_

However, I do not understand how it works

If I call CreateSpanAsync, what do I put in SpanName.FromProjectTraceSpan("[PROJECT]", "[TRACE]", "[SPAN]"),

My understanfing was to use ProjectId, activity.Id, "0000000" => still not showing the trace

Maybe I need to call BatchWriteSpansAsync ? I tried that too but still not showing the trace

=> Can you help or proide pointers ?

[amanda-tarafa]

First take a look at Distributed Tracing in Cloud Run documentation. A few things to note:

• In Cloud Run traces are stored automatically, but at a fixed sample rate of 0.1 traces a second. This rate is not configurable on Cloud Run. After leaving your application running for a long enough time, are you missing all traces or just most of them consistent with the sample rate? Do wait for a bit before checking, it might be that logs and traces are flushing at different times.
• Adding extra instrumentation (spans) will let you know metrics of specific code paths in your app, but it won't have an effect on the sample rate (and that's why adding spans is not showing the trace). If you only care about how long took the request overall, then you don't need to add spans.

If you want to trace more than what the Cloud Run sample rate allows, you'll have to store the traces yourself using Google.Cloud.Trace.V1.TraceServiceClient.PatchTracesAsync.

Alternatively you can use Google.Cloud.Diagnostics.Common and Google.Cloud.Diagnostics.AspNetCore3 (works for ASP.NET Core 3 or later) to instrument your code. But please read carefully these notes as they will affect how you need to configure the libraries.

[glb-cblin]

Hi Amanda

In Cloud Run traces are stored automatically

This is true ONLY for HTTP requests

Here, this is a pull subscription, meaning that there is no inconming http requests and so there is no trace => this is why I need to manually create one

Google.Cloud.Trace.V1.TraceServiceClient.PatchTracesAsync.

I dont understand this snippet (how am I supposed to start a trace with that snippet ?)

please read carefully these notes

Thanks for the pointers, I'll try "one trace per event"

[amanda-tarafa]

Oh, sorry for missing this was for Pub/Sub. So yes, you will have to instrument the code yourself.

The snippets we currently have are like code templates, they are not complete (we are working on it). On Google.Cloud.Trace.V1.TraceServiceClient.PatchTracesAsync:

• PatchTracesRequest.Traces is a collection of new or existing traces to patch.
• And here is a Trace as you would add it to this collection.
• You need to extract the trace context from the Pub/Sub message, it won't be set on the current Activity (we'll be working on this in the short to medium term).

If you look at https://github.com/googleapis/google-cloud-dotnet/issues/6367#issuecomment-903852079, step 3 shows you how to extract and parse the trace context from a Pub/Sub message. The rest of that comment, which is the conclusion of that whole thread, explains how to configure and use Google.Cloud.Diagnostics.Common for tracing non-ASP.NET Core contexts (as is Pub/Sub). You can decide to go with Google.Cloud.Diagnostics.Common or use Google.Cloud.Trace.V1 directly.

Let me know if you have more questions, happy to try and help further.

[glb-cblin]

Thank you, I'll try that and reopen if not enough