Move towards an OpenTelemetry style of trace observability

[chucklever]

As NFS (usually) involves more than one network host, distributed tracing might be a good fit for observing NFS observation in the field. The most common distributed tracing frameworks are based on OpenTelemetry. At the very least, OpenTelemetry style tracing provides a mechanism of associating disparate trace events into a DAG so that timing and other relationships can be visualized (as a tree or waterfall graph, for example).

If NFS client implementations adopt something similar and the NFS protocol itself is extended to exchange OTEL contexts, that would permit us to collect traces from multiple hosts at once and correlate them.

[chucklever]

As a starting point, implementing spans in NFSD might be a small but meaningful improvement (see Spans). We can build on the existing ftrace infrastructure that is already in the Linux kernel.

A span API might consist of:

struct span_contect *span_begin_root( ... );
struct span_context *span_begin( ... );

void trace_span_yada(struct span_context *sc, the usual trace arguments);

void span_end(struct span_context *sc);

where "span_begin" creates and initializes a span context, which is passed to trace points and span_begin calls that should be included in the span; and "span_end" captures the ending timestamp and puts a span_end event in the trace event log.

[chucklever]

One of the issues with true OTEL-compliant spans, though, is that they contain a variable amount of information. Much like netlink messages, they are self-describing and can contain zero-or-more of certain elements, unlike kernel trace points, which have a record format that is determined at compile-time.

For instance, a span can contain zero-or-more links to other spans. I guess we could accommodate that by using a dynamic array of links.