Rework trace storage

[rocallahan]

The current trace storage approach has some issues that we need to fix:

• Traces are written directly by the rr process and buffered data is not always flushed, so SIGKILL directed at the rr process (not uncommon, when rr is used as a wrapper around some program that normally is SIGKILLed by other code, e.g. on timeout) leaves an unusable trace.
• rr must finish writing a trace before anything can start consuming the trace. If we fixed this then you could start an rr replay concurrently with an rr record and be ready to debug as soon as an error is encountered instead of having to wait for the replay.
• There is no currently way to efficiently package a trace for copying or streaming to another machine for replay.
• Currently trace versioning is very primitive and there is no way for rr replay to consume trace versions other than the 'current' version.

On the other hand it has some important features we need to preserve:

• We avoid copying data by hardlinking and FICLONERANGEing files into the trace directory.
• Certain data records like memory-maps and trace-events are stored in their own files so tools can efficiently read those records without having to scan the whole trace.

[rocallahan]

FWIW I don't plan to fix this right away but I wanted to get the word out.

[rocallahan]

So what I have in mind here is something like this:

• Define an API for consuming a trace stream, similar to existing TraceReader. This will allow only reading a subset of records, e.g. just the task events. Most of the trace records can just be treated as binary blobs by this API, it's just for moving trace data around.
• Create an implementation of that API that receives data over an AF_UNIX socket. Pass filenames or file descriptors over the socket for the hardlinked and FICLONERANGE data. Mark event boundaries clearly in the socket protocol so a consumer can cleanly cut off a trace when the producer terminates unexpectedly.
• Have rr record push data into a socket using that protocol. We lose the shmem stuff we're currently doing but that's probably OK.
• Implement a "file-writer" consumer process that consumes from that API and writes the trace to a similar file structure to what we have now.
• Have rr record spawn a "file-writer" process to store the trace.
• Provide another implementation of the trace-consumer API that reads from the file structure (without going through a socket).
• Have rr replay and rr dump use that implementation via the trace-consumer API.

Then

• Create a library that consumes a trace via the trace-consumer API and pushes into the socket protocol.
• Use that to extend the "file-writer" process so that clients can connect to it via their own sockets and receive a copy of the trace stream (catching up based on what's on disk, if they connect to an in-progress trace).
• Create a variant of the file-writer process that stores the trace in a transport-friendly format: block deduplication and higher compression levels. This could be used as an alternative backend for rr record and as an independent tool that consumers regular trace files and produces the transport-friendly one.
• Create a variant of the socket protocol that does interactive block-deduplication instead of assuming access to the same filesystem, thus allowing streaming to remote hosts. Provide tools that wrap the trace-consumer API to export the data via this protocol, and to consume this protocol and produce the trace-consumer API.

[zouyonghao]

Hi @rocallahan What kind of signal should I send to stop the process safely if I can only send signals to rr?

[rocallahan]

I don't understand the question, sorry. What do you mean by "the process"?

[zouyonghao]

@rocallahan Sry, I mean the rr process. I have found that sending SIGTERM is fine.