Tracing TCP

[mjallday]

Background

• We are implementing tracing from the very edge of our platform.
• We support HTTP protocol as well as several TCP based protocols (e.g. SFTP, HTTP Tunneling, straight TCP streams etc).

Problem

We want to have all requests (HTTP and TCP) traced throughout our system from the edge. The TCP streams are encrypted. We cannot attach trace data to each request.

Proposal

We believe that there is value in exploring using a TCP header similar to PROXY Protocol to inject a TCP header.

The PROXY protocol provides a convenient way to safely transport connection information such as a client's address across multiple layers of NAT or TCP proxies. It is designed to require little changes to existing components and to limit the performance impact caused by the processing of the transported information.

The PROXY header is injected into the TCP packet and looks something like this

PROXY TCP4 192.168.0.1 192.168.0.11 56324 443\r\n

We want to propose something similar like this

TRACE b9c7c989f97918e1\r\n

We've seen other people ask if this is possible

• https://github.com/coredns/coredns/issues/624
• https://github.com/opentracing/opentracing-java/issues/292

I haven't seen a proposed solution yet.

Questions to address

• Is this a reasonable idea?
• Should this be proposed as a standard implementation similar to PROXY protocol?
• Does this work if we have a long-lived TCP connection?
• Does this correlate with https://www.w3.org/TR/trace-context/

[yurishkuro]

Are you sure referring to this as "TCP Header" is accurate? Afaik TCP Header structure is predefined by the TCP spec. What PROXY protocol seems to be doing is sending a specialized payload upon establishing a connection - a mechanism that is not fully transparent to the receiver, i.e. the receiver needs to actually support that or be confused by the incoming data stream. Please correct me if I am wrong, as I only briefly skimmed the PROXY spec.

Another issue is that tracing headers need to be sent on per-request basis, not just upon establishing the TCP connection.

[mjallday]

Are you sure referring to this as "TCP Header" is accurate?

probably not, that's just my attempt at using a familiar nomenclature. in the above link it's described as

"header" or "PROXY protocol header" is the block of connection information the connection initiator prepends at the beginning of a connection, which makes it the sender from the protocol point of view.

I'm not sure what term would work better but hopefully this conveys the concept accurately. happy to edit the description to something that's more technically accurate if there's a suggestion available.

i.e. the receiver needs to actually support that or be confused by the incoming data stream

this is correct. the receiver of the tcp stream needs to be aware of how to parse this.

Another issue is that tracing headers need to be sent on per-request basis, not just upon establishing the TCP connection.

yeah, I raised this as a question in the original post too. for our company this is an acceptable tradeoff. using http connect tunnels as an example, we could establish the tcp trace id as the parent trace id for the tunnel and then the requests within that tunnel could be treated separately or as children.

I definitely understand there's some issues to solve or trade offs to accept to make it work, do you know of any other strategies for achieving this? We deal mostly with encrypted data and a lot of non-http payloads so parsing the payload and injecting a header isn't always an option.

[yurishkuro]

You can try it, but I don't think the mechanism is generic enough, especially given one-per-connection limitation that's not going to work for many users.