Incorrect trace with envoyproxy

[BatanGaming]

In my app I have 2 envoyproxy gateways with enabled tracing and some web services. When I do request to web service I expect to see one trace with 3 services but instead I get 2 traces - first with 2 gateways and second with web service. First trace: Second trace: What's wrong with it? envoyproxy uses zipkin tracing. Maybe there's some issue with it?

[Falco20019]

It looks like there is some disconnect between the traces. Maybe not all systems are configured to use Zipkin? All need to use the same system, otherwise the trace IDs might not get picked up.

[BatanGaming]

For now there are only 2 gateways (which are configured to use Zipkin) and one asp.net core web application which is using this jaeger client. So, do I need to configure asp.net core app to use Zipkin? If so, how to do it? Now it's configured to send traces to http://jaeger:14268/api/traces (which I guess standard path for http traces) and nothing else.

[Falco20019]

Either set the environment JAEGER_PROPAGATION to b3 or when using the Configuration helper, use WithCodec to give it an instance of CodecConfiguration configured using WithPropagation(Propagation.B3).

Configuration.SenderConfiguration.DefaultSenderResolver = new SenderResolver(loggerFactory)
    .RegisterSenderFactory<ThriftSenderFactory>();
CodecConfiguration codecConfig = new CodecConfiguration(loggerFactory)
    .WithPropagation(Propagation.B3);
Configuration config = new Configuration("myServiceName")
    .WithSampler(...)   // optional, defaults to RemoteControlledSampler with HttpSamplingManager on localhost:5778
    .WithReporter(...)  // optional, defaults to RemoteReporter with UdpSender on localhost:6831 when ThriftSenderFactory is registered
    .WithCodec(codecConfig);

ITracer tracer = config.GetTracer();

See https://github.com/jaegertracing/jaeger-client-csharp/blob/master/README.md for how to configure the tracer correctly.

[BatanGaming]

Thank you very much. I did this configuration and everything works fine

services.AddSingleton(provider => {
    var loggerFactory = provider.GetService<ILoggerFactory>();

    var senderConfiguration = new Configuration.SenderConfiguration(loggerFactory)
        .WithSender(new HttpSender("http://jaeger:14268/api/traces"));

    var reporterConfig = new Configuration.ReporterConfiguration(loggerFactory)
        .WithSender(senderConfiguration);

    var codecConfig = new Configuration.CodecConfiguration(loggerFactory)
        .WithPropagation(Configuration.Propagation.B3);

    var samplerConfig = new Configuration.SamplerConfiguration(loggerFactory)
        .WithType("const");

    var config = new Configuration("web api", loggerFactory)
        .WithReporter(reporterConfig)
        .WithSampler(samplerConfig)
        .WithCodec(codecConfig);

    var tracer = config.GetTracer();

    GlobalTracer.Register(tracer);
    return tracer;
});

[BatanGaming]

Hi again, I have a few more questions:

1. How properly disable tracing for Entity Framework?
2. I added some decorator for MediatR request handling and then in UI I got invalid parent span IDs, so span's order is incorrect (for ef core spans' order is incorrect too) (last span should be child of first span);
3. What does warning clock skew adjustment disabled mean and how to disable/fix it?

Decorator's code:

var requestName = request.GetType().Name;
using var scope = _tracer
    .BuildSpan($"handling-{requestName}")
    .AsChildOf(_tracer.ActiveSpan.Context)
    .StartActive(true);
TResponse response;
try {
    scope.Span.Log($"Handling {requestName}");
    response = await next();
    scope.Span.Log($"Handled {requestName}");
}
catch (Exception ex) {
    scope.Span.Log(ex.Message);
    scope.Span.SetTag(Tags.Error, true);
    throw;
}

return response;

[Falco20019]

One question upfront: Are you using ASP.NET Core? If yes, you might want to use https://github.com/opentracing-contrib/csharp-netcore in combination with this library. I assume you already do something like that, otherwise, entity framework should not log anything.

You don't have to add .AsChildOf(_tracer.ActiveSpan.Context) since this is done automatically and handled correctly if it's the first span in a trace. When using StartActive, you must ensure, that the order is kept when disposing. This is only relevant when doing multi-threading. Use an ScopeManager to distinguish different scopes.

clock skew adjustment disabled comes from the adjuster in go and is not part of this lib: https://github.com/jaegertracing/jaeger/blob/master/model/adjuster/clockskew.go#L57 Maybe @yurishkuro can tell you when this happens. But I assume it is related to your issue with span ordering.

[BatanGaming]

Thank you very much for your reply.

1. Yes, I'm using ASP.NET Core and now I understand what AddOpenTracing() for, thanks for explanation.
2. I put it incorrectly. At screenshot it's not the first span in a trace. So, I removed .AsChildOf(...) and nothing's changed. I can't quite understand how to use ScopeManager in this case.

[Falco20019]

I did something similar here: https://github.com/opentracing-contrib/csharp-grpc/blob/master/src/OpenTracing.Contrib.Grpc/Handler/InterceptedServerHandler.cs#L69-L84

https://github.com/opentracing-contrib/csharp-grpc/blob/master/src/OpenTracing.Contrib.Grpc/GrpcTraceLogger.cs

You might want to call Finish on the span to terminate it right where you want. This ensures the flow is as expected, independent of when the Dispose is done.

For better readability, you can extract scope.Span into a variable, the instance will stay the same for the whole scope. So the property access is not necessary for each call.

[BatanGaming]

Ok, I'll try, thank you very much

[Falco20019]

@BatanGaming In #209 we had the same issue. Finishing the span and disposing the tracer on shutdown solved the problem. If not disposed, it can happen that the span is not yet sent and the application stops prematurely leading to the loss.

[BatanGaming]

Oh, yeah, I've tried this already and it works. Thank you and sorry for not notifying you