Implements experimental tracy client multiplexer

cipharius commented 6 months ago

This is a proof of concept implementation of tracy multiplexer which allows tracing multiple Tracy clients in realtime. It intends to resolve issue #471.

The multiplexer works by listening for client broadcast messages and automatically establishing connection with Tracy clients. Once connection is established between multiplexer and a Tracy server, all client events are fused into one event stream and passed on to the server.

Usage:

Launch the multiplexer
Launch at least one client
With Tracy profiler connect to multiplexer over TCP port 8085
During profiling session, new Tracy clients will be automatically adopted and included in the timeline

Some implementation details:

The multiplexer steals first client's welcome message and uses it to establish connection with the Tracy server;
Different client events are combined at ThreadContext boundaries;
Server requests are replicated to all connected clients, once all clients have responded to the request the most appropriate response is picked and forwarded to the Tracy server. This part is lacking in this proof of concept and works best with 3+ clients, because in that case it's possible to find which response differs from rest and use that;
Time is normalised and adjusted within multiplexer in order to have a coherent timeline in the Tracy profiler;

This proof of concept has some fragile parts that would be challenging to keep up to date as the profiler is updated. To address this, multiplexer, TracyWorker and TracyProfiler should be refactored to increase procedure reuse. For example I had to reimplement procedure that computes event packet size and had to manually sift through all events that contain time.

Arpafaucon commented 3 months ago

Hi ! I set up the env today but I cannot reproduce your example :/ When spawning the different processes in the order you specify, I get the following error

❯ ./multiplex/build/tracy-multiplex
Starting Tracy multiplexer on port 8085
Listening for client UDP broadcast messages...
Error: Multiplexer has no welcome message, launch the first client
[... repeated multiple times]
Error: Multiplexer has no welcome message, launch the first client
Connecting to a client:
    Name: tracy-test
    ID: 34729646139584
    Address: 192.168.56.32
    Port: 8086
Welcome message:
    timerMul: 0.434029
    initBegin: 13107256857726
    initEnd: 13107820545094
    delay: 191
    resolution: 56
    epoch: 1719930553
    exectime: 1719930161
    pid: 89027
    samplingPeriod: 100000
    flags: 4
    cpuArch: 2
    cpuId: 34540
Connected to client successfuly

Got supposedly weird type 59
tracy-multiplex: /home/gregoire.roussel/dev/perf/tracy/multiplex/src/multiplex.cpp:736: bool handle_client_response(ClientConnection &, const tracy::QueueItem *, uint32_t): Assertion `ev->hdr.idx >= (int)tracy::QueueType::StringData' failed.
[1]    88859 IOT instruction (core dumped)  ./multiplex/build/tracy-multiplex

the assertion is in the default case of your big switch handling client responses.

Notes

the Got supposedly weird type 59 is from me, to understand what's the hdr.type that's causing the failed assertion.
59 is a tracy::QueueType::ThreadContext; I guess the assert documents that we should not expect ThreadContext messages there ?

cipharius commented 3 months ago

@Arpafaucon Are you testing with tracy GUI and client compiled from this commit's checkout?

It will most likely fail if you try to use it with too new tracy version, due to how coupled the code is with the protocol.

Arpafaucon commented 3 months ago

@Arpafaucon Are you testing with tracy GUI and client compiled from this commit's checkout?

It will most likely fail if you try to use it with too new tracy version, due to how coupled the code is with the protocol.

yes, I compiled capture, multiplex and my test program from your commit 57fe7556ec68587acb62aca40b7865647eaec9f3

wolfpld / tracy

Implements experimental tracy client multiplexer #766