gRPC call requires span in context passed from client for server trace to work?

[dabfleming]

So this may be working as intended but bear with me a moment...

Was trying to add tracing to a simple SayHello style gRPC service with both client and server in Go. I added all the necessary code to the server and then spent some time trying to figure out why I wasn't seeing any traces being exported.

After playing with github.com/rakyll/opencensus-grpc-demo a bit I realized that tracing only worked on the server if the client looked like this...

ctx := context.Background()
res, err := client.SayHello(trace.WithSpan(ctx, span), req)

...but not if it looked like this...

ctx := context.Background()
res, err := client.SayHello(ctx, req)

As I said, this may be working as intended but it seems odd to me that the server tracing only works when the client passes in a span. Shouldn't the server create a span regardless of a parent span being passed from the client? Maybe not and I'm fundamentally misunderstanding this.

[bogdandrutu]

@dabfleming I agree, if the client does not send a context but the server is configured with tracing the plugin should start a Span on the server side as a root of the trace.

[rakyll]

@dabfleming, what's your sampling rate? The server should start spans even if the client doesn't have any root spans created.

[rakyll]

@dabfleming, I modified the demo to remove the root span and increased the sampling rate. You can see that traces are started even from a context.Background(). See https://github.com/rakyll/opencensus-grpc-demo/blob/master/client/main.go#L70.

[rakyll]

@dabfleming, when in doubt, you can also use zpages to investigate what's going on in the process.

For example, add the zpages handlers to the default mux in main.go

import "go.opencensus.io/zpages"

go func() {
    // Serve the prometheus metrics endpoint at localhost:9999.
    http.Handle("/metrics", prometheusExporter)
    zpages.AddDefaultHTTPHandlers()
    log.Fatal(http.ListenAndServe(":9999", nil))
}()

Then you can see the traces and RPC stats via http://localhost:9999/tracez and http://localhost:9999/rpcz.

[odeke-em]

See https://github.com/rakyll/opencensus-grpc-demo/blob/master/client/main.go#L70.

For posterity, here is a permalink to the sampler change that @rakyll mentioned above https://github.com/rakyll/opencensus-grpc-demo/blob/bc6bd99a1a56eb9bc26e8f844dc2c3754342041e/client/main.go#L67-L68 as the quoted line number has been changed.