Thrift.UdpSender not sending Span even though it's under MaxSpanBytes

[js8080]

Requirement - what kind of business use case are you trying to solve?

I am attempting to send Spans to the Jaeger Agent via the Thrift.UdpSender that are over 65000 bytes.

Problem - what in Jaeger blocks you from solving the requirement?

I have my Jaeger Agent configured to accept Spans that are quite large via the following Agent CLI parameter:

--processor.jaeger-compact.server-max-packet-size=1048576

I then create a Span in my ASP.NET Core code and call _tracer.ActiveSpan?.Log() to log some fairly large data to the Span. This results in a Span that is over the default 65000 bytes limit for the Thrift.UdpSender so when I instantiate my UdpSender, I pass in a higher value for the maxPacketSize parameter (e.g. 1048576 as shown here):

    var sender = new Jaeger.Senders.Thrift.UdpSender("localhost", 6831, 1048576);

    var reporter = new RemoteReporter.Builder().WithLoggerFactory(loggerFactory).WithSender(sender).Build();
    var initialSampler = new ConstSampler(true);
    var sampler = new RemoteControlledSampler.Builder(serviceName)
        .WithLoggerFactory(loggerFactory)      // optional, defaults to no logging
        .WithInitialSampler(initialSampler)    // optional, defaults to ProbabilisticSampler(0.001D)
        .WithPollingInterval(pollingInterval)  // optional, defaults to TimeSpan.FromMinutes(1)
        .WithSamplingManager(samplingManager)  // optional, defaults to HttpSamplingManager("localhost:5778")
        .Build();

Unfortunately, even though my size limit is set to well over 65000 characters and I can see sender.MaxSpanBytes = 1048543, my Span is still not sent to my Agent. I can tell this because I have my Agent running with debug level output and if I decrease the size of the Span enough, I see those being sent in the Agent logs (but not the larger ones) -- for example:

jaeger-agent_1  | {"level":"debug","ts":1608831249.214059,"caller":"processors/thrift_processor.go:120","msg":"Span(s) received by the agent","bytes-received":63662}
jaeger-agent_1  | {"level":"debug","ts":1608831254.2172835,"caller":"processors/thrift_processor.go:120","msg":"Span(s) received by the agent","bytes-received":64677}

So it seems even though I am specifying a larger packet size limit, it is still not being respected. I also have the log level set for my .NET Core application to log Trace level output for Jaeger and Jaeger.Senders namespaces but I see no log output to the console for these Spans:

  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "OpenTracing": "Debug",
      "Jaeger": "Trace",
      "Jaeger.Senders": "Trace",
      "Microsoft.AspNetCore": "Warning",
      "Microsoft.Hosting": "Warning"
    }
  },

Strangely, I tried setting the max size limit to a much lower value to see if I could get the ThriftSender to log this exception but I still am not seeing that:

throw new SenderException($"ThriftSender received a span that was too large, size = {spanSize}, max = {MaxSpanBytes}", null, 1);

So I can't tell where the Span is actually being dropped. Am I doing something wrong with my sender setup / DI injection here that would reveal these logs and help me troubleshoot myself or is something else going wrong like the maxPacketSize not being respected? I looked through the jaeger-client-csharp code and I am not seeing an obvious issue.

Proposal - what do you suggest to solve the problem or improve the existing situation?

Respect the maxPacketSize passed in to UdpSender.

Any open questions to address

[js8080]

After further investigation and debugging the code from source, I see the reason why I am not getting any logger output when my Span size exceeds the configured limit but still not sure why larger Spans that are under the limit are not going out.

In Jaeger.Reporters.RemoteReporter.ProcessQueueLoop the SenderException is caught and recorded the dropped span count metric but not written to _logger:

        private async Task ProcessQueueLoop()
        {
            // This blocks until a command is available or IsCompleted=true
            while (await _commandQueue.OutputAvailableAsync())
            {
                try
                {
                    var command = await _commandQueue.ReceiveAsync();
                    await command.ExecuteAsync();
                }
                catch (SenderException ex)
                {
                    _metrics.ReporterFailure.Inc(ex.DroppedSpanCount);
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, "QueueProcessor error");
                    // Do nothing, and try again on next command.
                }
            }
        }

[yurishkuro]

I am attempting to send Spans to the Jaeger Agent via the Thrift.UdpSender that are over 65000 bytes.

Not recommended. Even on local host network packets larger than 64k can be fragmented, and the Jaeger agent was never designed for or tested to handle that.

[js8080]

Not recommended. Even on local host network packets larger than 64k can be fragmented, and the Jaeger agent was never designed for or tested to handle that.

If I bypassed the Agent and sent straight to the Collector via HTTP, would that allow me to send larger Spans? I don't really want to do that but I may need to support larger Spans.

[yurishkuro]

Yes, you can do that but not with UDP. This SDK supports submitting spans via grpc/proto, I recommend using that.

[js8080]

OK, I will explore those options if I need them. What should we do to address the issue documented here though? The Jaeger C# Client has a max packet size setting but then doesn't seem to truly respect it or at least going above 65k bytes does not work. Should there be a change to code a max value for the setting so people are not mislead by the availability of this option?

[yurishkuro]

What do you propose?

[js8080]

Would a documentation update actually be sufficient?

[yurishkuro]

It's a start. It might have been a mistake to expose max packet size as configurable.

[js8080]

Should the setting just be deprecated then? I don't know how painful that will be to break backward compatibility.

[yurishkuro]

I also don't know if/how people are using this config option. I wouldn't deprecate it, but we could add a warning to the description "not recommended to exceed 64KiB to avoid packet fragmentation".

[js8080]

That seems reasonable to me.

[ttanga]

Could be better to adjust span and try to calculate size again instead of throwing here

  ThriftSpan thriftSpan = JaegerThriftSpanConverter.ConvertSpan(span);
  int spanSize = CalculateSpanSize(thriftSpan);
  if (spanSize > MaxSpanBytes)
  {
    throw new SenderException(
      $"ThriftSender received a span that was too large, size = {spanSize}, max = {MaxSpanBytes}", null, 1);
  }

eg remove/truncate logs and add some specific tag

[Falco20019]

Sorry for the late response. I‘m currently out of office and will have a look next week when I‘m back.

[Falco20019]

I will limit the max value. Usually, this should not be changed by the user (only to limit it further down on problems).