Cali0707
commented
6 months ago cc @pierDipi
Open mauricecamphadrian opened 6 months ago
Cali0707
commented
6 months ago cc @pierDipi
pierDipi
commented
5 months ago /assign
pierDipi
commented
5 months ago Send 20 events with random EventId and random traceparent headers to the knative-eventing broker ingres (http://kafka-broker-ingress.knative-eventing.svc.cluster.local/default/broker)
@mauricecamphadrian do you have a script I can run to do that or would you mind providing one please?
mauricecamphadrian
commented
5 months ago I used a small dotnet tool to generate the events since our producing service is also a dotnet service:
Program.cs:
using System.Collections.Concurrent;
using CloudNative.CloudEvents;
using CloudNative.CloudEvents.Http;
using CloudNative.CloudEvents.SystemTextJson;
const string RequestURI = "http://localhost:8080/default/broker";
const int WorkerParallism = 10;
const int RequestsPerWorker = 2;
var bag = new ConcurrentBag<KeyValuePair<Guid, string>>();
await Parallel.ForAsync(0, WorkerParallism, async (i, _) =>
{
var client = new HttpClient();
for (var x = 0; x < RequestsPerWorker; x++)
{
await MakeRequest(i, client);
}
});
foreach (var (k, v) in bag)
{
Console.WriteLine($"Sent event with id {k} and traceparent {v}");
}
async Task MakeRequest(int workerIdx, HttpClient client)
{
var eventId = Guid.NewGuid();
var cloudEvent = new CloudEvent
{
Data = new(),
DataContentType = "application/json",
Id = eventId.ToString(),
Source = new Uri("tester", UriKind.Relative),
Time = DateTimeOffset.UtcNow,
Type = "test.event"
};
var traceparent = GenerateRandomTraceParentHeader(out var traceId, out var parentId);
var content = cloudEvent.ToHttpContent(ContentMode.Binary, new JsonEventFormatter());
content.Headers.Add("traceparent", traceparent);
content.Headers.Add("ce-partitionkey", $"{workerIdx}");
var response = await client.PostAsync(RequestURI, content);
if (!response.IsSuccessStatusCode)
{
Console.WriteLine("[WRN] Unexpected status code received");
}
bag.Add(KeyValuePair.Create(eventId, traceparent));
}
static string GenerateRandomTraceParentHeader(out string traceId, out string parentId)
{
var traceBytes = new byte[16];
Random.Shared.NextBytes(traceBytes);
traceId = Convert.ToHexString(traceBytes).ToLowerInvariant();
var parentBytes = new byte[8];
Random.Shared.NextBytes(parentBytes);
parentId = Convert.ToHexString(parentBytes).ToLowerInvariant();
return $"00-{traceId}-{parentId}-01";
}Project file:
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net8.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="CloudNative.CloudEvents" Version="2.7.1" />
<PackageReference Include="CloudNative.CloudEvents.SystemTextJson" Version="2.7.1" />
<PackageReference Include="Microsoft.AspNetCore.Mvc" Version="2.2.0" />
</ItemGroup>
</Project>I spent some more time trying to find where this seems to be going wrong and I think it happens in the event sending logic in WebClientCloudEventSender.send():
VertxMessageFactory.createWriter(req).writeBinary(event) in results in correct trace ids.onSuccess callback of writeBinary() results in an incorrect trace id that matches the trace id as seen in the kafka topic.I tried to dive deeper but I'm too unfamiliar with java and Vert.x.
mauricecamphadrian
commented
4 months ago Any update on this?
Describe the bug Trace ids appear to be mixed up by some knative component. We roughly have the following setup:
For each event sent by AppA a new root trace is started with a unique trace id. This is reflected in the
traceparentheaders of the events we see in the backing topic for the Kafka broker. However the trace ids we receive in AppB get mixed up according to our observability platform, many spans are being grouped together under a few trace ids. This is also reflected in the backing topic of the Kafka channel, which leads me to believe that some KNative component is causing the issue.Expected behavior We expect each event to retain its original trace id as it propagates through Knative components.
To Reproduce I have managed to create a minimal example to reproduce this:
As you can see from my results there are several events that were received with trace id
1204504a7b59e014605a9a9ee67cb63abut only 1 was sent with this trace id.Knative release version 1.14.1 Reproduction was done using commit 28b46159cf8fe6877756459198c0a8f49c2d6626