This PR changes the representation of OTel Traces in Apache Arrow to improve the overall compression ratio.
Switch from a fully hierarchical Arrow representation to a hybrid representation to better enforce the RecordBatch order and improve compression efficiency. The ResourceSpans, ScopeSpans, and Spans are represented hierarchically in a single RecordBatch. All attributes are now represented in their own RecordBatch and sorted globally. There is a RecordBatch for each attribute family (i.e. Resource, Scope, Span, Event and Link attributes). Events and links are also represented by a dedicated and related RecordBatch, sorted on specific columns. This hybrid representation offers better performance because the order can be applied globally and more specifically for each type of linked data.
Delta-encoding is used to represent attribute ids and internal span ids in Resource, Scope, Span, Event, and Link object. Since these identifiers are intentionally ordered globally and monotonically (increasing) the delta encoding cancels out the space occupied by these identifiers on the main entity side of the relationship.
Fix an issue in the profiler. The improvement indicator was calculated on the P99, which made results not representative of a normal behavior. The improvement indicators are now calculated on the average.
Use Arrow Duration data type to represent the difference between StartTimeUnixNano and EndTimeUnixNano. EndTimeUnixNano is reconstructed on the consumer side based on the duration and StartUnixTimeNano. This encoding provides a better compression ratio because the duration can be encoded efficiently with a dictionary.
New set of stats can be collected on traces to better understand usage and compression issues. The flag -stats can be used with the traces_benchmark command.
Limits:
The maximum number of Resource per batch is 65535.
The maximum number of Scope per batch is 65535.
The maximum number of Span per batch is 65535.
The maximum number of Event per batch is now 4,294,967,295
The maximum number of Link per batch is now 4,294,967,295
The number of attributes is not limited (for any type of attributes).
This PR changes the representation of OTel Traces in Apache Arrow to improve the overall compression ratio.
RecordBatchorder and improve compression efficiency. TheResourceSpans,ScopeSpans, andSpansare represented hierarchically in a singleRecordBatch. All attributes are now represented in their ownRecordBatchand sorted globally. There is aRecordBatchfor each attribute family (i.e. Resource, Scope, Span, Event and Link attributes). Events and links are also represented by a dedicated and relatedRecordBatch, sorted on specific columns. This hybrid representation offers better performance because the order can be applied globally and more specifically for each type of linked data.Durationdata type to represent the difference betweenStartTimeUnixNanoandEndTimeUnixNano.EndTimeUnixNanois reconstructed on the consumer side based on the duration andStartUnixTimeNano. This encoding provides a better compression ratio because the duration can be encoded efficiently with a dictionary.-statscan be used with the traces_benchmark command.Limits: