New arrow data types

[djaglowski]

Is your feature request related to a problem? Please describe. The discussion on https://github.com/open-telemetry/oteps/pull/171 included some comments (example) about potentially needing to define a new signal type in the collector, in order to support columnar encodings. I'd like to offer a more specific proposal for how this should be handled in the collector.

Describe the solution you'd like For each current data type supported in the collector, introduce a corresponding arrow data type. Names here are only meant as placeholders, but let's say atrace, ametric, and alog.

Introducing three distinct data types means that we could use the collector's pipeline system to our advantage.

• Each type of component opts into each data type independently. Therefore, support for each arrow type can be added as necessary.
• Pipelines are always associated with exactly one data type, so this would allow for the construction of arrow-specific pipelines. Validation of a pipeline enforces that all components within the pipeline support that data type.
• Connectors support ordered pairs of data types. Adding new arrow data types would allow for a connector that supports translation between pdata and arrow formats. (Phase 2 adaptation layer). Consequently, this allows data of either type to move through processors and exporters that only support one of the pdata or arrow formats.
  • e.g. arrow receiver -> arrow processor -> to_pdata connector -> pdata processor -> to_arrow connector -> arrow exporter

[jmacd]

Thanks - I like this proposal.

[lquerel]

I really like this approach, as it brings us closer to optimizing the collector for transport and processing, enabling an optimized end-to-end Arrow representation without breaking the existing ecosystem.