Columnar encoding for the OpenTelemetry protocol

[lquerel]

Ongoing experiments, research, and draft specification for a multi-variate columnar encoding of OTLP.

At the request of @jmacd

[linux-foundation-easycla[bot]]

The committers listed above are authorized under a signed CLA.

• :white_check_mark: login: lquerel / name: Laurent Quérel (95da4aea60cf6644c5f4827df7f7fead0cace449, aec7656ca4c0f21f03d905d5a1183e68d3a76c7d, a3fbdb730fac42e834ac8c6e1303c8efb76f8a17, 6b550f4e09f211463d2dd162159d7db50dca1f01, 3496ca11caa5b86a15d7baf76dc77661ac7a5ba8, 56b3357c61e60997d2bf5c193889fd75eadd3667, d760154fe981d566299fff838c1f6e70a3a2a356, 589dc8b3b012bfaf9a04032bc60809fc256955bb, a51696224f325ab63cafe2bb6b7ac03dad8072d5, ecf45d2062ab73bd0dd0fde4ab2368b4af8d97b4, 58e9a5363e14c884a05454596c43411e47a65e00, c809b1be52a9bd72408970d2f0d34b1019f8d81d, 150d987df98acc54feb6318428b17addb4380553, 43340eaedcee3ae0ba0e495a94f8b6dd24019908, 1448c911301088bbd10e5a4dadf7a5e029c41aac, 0db08ec3c1bd945e0e219b34967ede302108e2bc, f3739cf5ab591ef7bc9e6ff21694428e4d438461, 2218179b994985abc3e9032de38051502a4dc364, ec813fec7739d163a31a28de9b146ba08f79aa05, 8c34a9153b520dabdf93904e2a05bb22efc80412, e3c38db3d607cac160c05833261b0a51383bb492, 68b4ed6635a68c69140a17e9de1f98bdf0e4fdc6, e25b3c60586f3e6b3f635ebc60d11ea2daea9098, 07714b11d5d0fd735831634a0db999c5a9128e4b, e82db1a8afe79482d8a9c0a1e3c17e86fe09ab96, d9a621a8baafe6fa7f4c9a587017dbfcc9701810, 8c7333e31a6fcfdc396049b80107077cebf20b63, 91f9561a54db95e1831eaadacebdc489b6f6dca3, c58a4ade6fb66ab6208b1dfcfbc3686afe0ded42, 52a85f77762ef61a0bcb747b5a6c2f572990b63c, 0b3f02d6514a6c0a0ccf5e6ef6159e93e37e32df, e8c6ef11862738cdc2c12f0af81210965d4f0487, dc74dd39a07bf905dbd26e99820206b689730cf1, fffce3353f238666772d2c74521f94c919c1fc06, 72768c4f01ad5db9f365d0ef1f6fc37d218af42c, 869559197d5569d633a5f54ea5bcbf7098f0c7f5, 9139969285083d70b8424d98a13ace3551b3164e, 6499c63b36646eb2f093294c34060c7bccb3578a, aa9c513ba708544a3652348a562c8131d79aa843, f0c6b90daf64b4fd323b7de77ab3140e63c1ccd7, fce05afe6a9df4351fa1f15ea8499b91d86010f7, 33ea57d6b441b719dac2d85d3bccfffb04553cc1, 3d636d57065a7ed1f4a3d48d80089b6f989d77cb, 19653e1ce6b4f30c609a3e01de3cb46ced66016a, 0590cb852727c3a2a9f8e76aba57dd29d061b2da, 8449b67828571dc0525a4fab9a2f1f66c277a6df, 0be8ae33d164df7ca9207c612bf9d636490dc9cd, ab93d23e6ce3b93a1ce68fe0d725cf9c6e05f6a9, f48c9522acf743f0e580446f625478600a460fae, 0e6b9ff6b3f727cfba4da9fd77b2bb1df44a4710, 577b4bd4575eef5a32d63ebcaf2bd9cd416c3104, 5b62001c42c926850f735f355f4ca4b747538afb, ec0a62391cd962cf8248224e31e242e791fb99ff, c0d134d25746baca34ad06af1bddb2a2ad02cc2c, 4f311f87a6e73afc8b331f08504558720059539d, 4baab68856107f6bb3431305a5a24924f5c8e815, 6b3d916a9dd171c462b3bb24147b8382bab55f6e, 3cc34584b57ff602df4b5b3c7942aaa461e515cd, f24cb9cabd58806f6f06bcb60fc284c60ee536cb, efa0aad8b7bcc803a78d69e0d887c2374c068bcb, b8cbf5564e90dac31df9f6e05bfec7affcda83a7, b3b00c0a832ec63ee5ba9a65e3fecb8193ffe5aa, b4a95df94a496053c007ab33fcbfb8172f2c3de2, 8271a6567428157e2369ab564ed66721f95099a8, 50557bc75df8f7ea830f813327e076267e43f302, e0dae9943648070c4ea45dd9aa5457e67c97f3b3, 321bbc582ab0b1968e5b117f4cf2afacc5fd0084, 50d4cd392c49c238c7995819785a224816fb2f5d, 5f1e2d8871b1993a5f8a4e2ef075ce8fbfd7f10e, 9e38c4736bee7afe144d2d973f6dad9582c20753, 44b2743a54dde56059f42817fbbe0b45bc68523d, 198f114f2121d14ccd7e0e2196ed69306343a673, 778cf0aa05e63bf7b857baee54ffb0d42034a1c2, e520b036d004a2c0c59e2011425424601d1672ea, 3952c527205a8182a9fb60fd4e3a8293554acdad, 21cd6651b6ea4b191e94f626f9e50001ade71068, ef2c09f0d02f724795515bbce1cd8ed980cebbaf, ed78a511cd09c32f0c9b76fdf8f349ba87e32bde, 1524d84a072f16bc3d342ee60f1d6f926368d655, f0dc75325684ed5b85e211bddee0228b604483b4, 8e65c10c78dc44fea72b47b272282979943f9d3d, 3f73d39af66b63d44084e437b713873983afb1fa, c78c78efd9aab18c83244f99e5815d506e9f932c, 20b4e308fbc8aca62b09bcf85849f44d117e5943, c5c747caf136c2295bab35d6435a39a1be3ca393, 87b535264d83f5d25edd5087faffcf4e050e4433, d648aba156e2521961866b14f55c1b6e2b39b9ea, 942c98e2aa06cfba2ed27a532604787c77168333, d5e6a89d37a3dd2fc992fd904c7b35df89d30099, 4a10df3227402c9c27aa0c61a091ca86ca0c0085, f0dd6389897145418d6425989f001a9d5767d043, 1f0e9fc49081d101927e80d22c4eca9af56f7e54, 4454d02138ee6e7218cb2c49a40a3a2cbfca4fb9, efeb9e943a62fe708507005a19f7bf7c7f0726c8, eade6b5234f3d09208dcdccabad3f0bff40a2a6d, e6f7dc6e9e1586db6bed03a9db1b9ae84e5adcd7, 1fde1082a588ca9641fc2681179bd3697edd5d68, 8a2b7135b9289051908055cc1e9fbfbbf154785a, 686467c8e1df99366cf2293cc580b77bce7d9563, e08d43d6eff87e735adaa07c9d2fab82cb9b9a00, 110990ba8b65746fd61eb387262828aa562732b0, b9e84522974adc628bc71d0b9df55f7a7f0a1c53, 6786b8c6e3df8d2b3999903c466b49fc9292e806, df286554fc53f04f7c851a097388e343d9ceb06a, e73155df136f2584882f3b8e0af398d608c2b22e, 84dac539756b2f96810681eb4bbc198af8760e26, 1b78ba689e66b3a924b2f400dd9d74bc2bbbc4bf, 8f82586061504365568729104630c397c098be46, 72fbebacc266935cf268ba63ec61fb7c54ec606b, fb17f4bd74457ae674ec415ed448fe78fcd491a2, 2aa20479090575ebbbb6024a27f0d802d49d1b21, 21bd7cb8e6c78afccb2c4def41a3c9b15e2184f2, 621475f52acae81112b890659a241d075ab4a362, 4bed578d662b1e5827c011e88618de7653cb080b, 09749cc07d04c9a92cc2f52f6882d31fbb4159b6, 549061c48352ab463ba0325d20d7f05837db87ac, 64f4a1d908b97808a00adeeb71f52a796e908866, ea6fd96b7d2e8c8f0e62d44a46dbbe1e8c703426, ec7736858725e2dc8108d5fb89f2aa2c9b00d1dd, 6c2e9b5d6fd54e376327337215f075a27fe5d2c3, ce274de035b6d4f6d7af2947f7352ac141c9729f, df06cea421b00b2f5baad67131660f0ba4eb8082, f83a3788f69ccd53222f0f7a69055acbef3d120b, 9bdeab28efe2fa3cf347470651139ebad858352a, d5e8966542989d72795f7dad74cd029d5466755f, fae225a87a0f532ba18b6d546fbd8133e160b881, 1892d310148e986374a3a125bfa369b015ad846b, 55122642991e190629568dccb89ba85d1b8122f2, 560ca64666dd94092949ca362f9b8870fdb687bb, 9f68a84e0f99ad680b657658c9e0600eb61d6f84, 34fd4df6c50a04c5278bea4adc099dadb3d5b89f, 25bf657737d6dc070725687ad7175c84f58fd083, 7fbd2693524f86edc59946141d52ef77028fc717, 0c3742d8796ec94170d2834e6084a305dfdbd0c4, 0ffddbf601dc12aa0fc2c9d25aae2052b17d5014, e5c64e968245af32fd60b97953c5374c32be915d, 0f1faf1016d04df57b0f438997354a9b474e9ce9, f26c725236e9e1896f397100035f446093d31f8a, ad0669385b7eb3831e6a713b07fdc3c0eaf71a06, a924a99177bcd549180bb3c00ce7b74db6e736e1)
• :white_check_mark: login: carlosalberto / name: Carlos Alberto Cortez (81eb951aed85751b9e04773b812cb68277aa5162)
• :white_check_mark: login: atoulme / name: Antoine Toulme (55122642991e190629568dccb89ba85d1b8122f2, 560ca64666dd94092949ca362f9b8870fdb687bb, 34fd4df6c50a04c5278bea4adc099dadb3d5b89f, 25bf657737d6dc070725687ad7175c84f58fd083)
• :white_check_mark: login: reyang / name: Reiley Yang (0c3742d8796ec94170d2834e6084a305dfdbd0c4, 0ffddbf601dc12aa0fc2c9d25aae2052b17d5014, e5c64e968245af32fd60b97953c5374c32be915d, 0f1faf1016d04df57b0f438997354a9b474e9ce9, f26c725236e9e1896f397100035f446093d31f8a, ad0669385b7eb3831e6a713b07fdc3c0eaf71a06, 5bd1d003e287b16a5e3e9dcad740ea81a46ee92d)

[tigrannajaryan]

Closing/reopening to see if EasyCLA passes.

[lquerel]

Closing/reopening to see if EasyCLA passes.

@tigrannajaryan I have been able to fix the CLA issue by adding my professional email into my Github account.

[lquerel]

@tigrannajaryan and @yurishkuro is there anything I can do to complete/clarify this OTEP?

[tigrannajaryan]

@tigrannajaryan and @yurishkuro is there anything I can do to complete/clarify this OTEP?

@lquerel I may have missed it but I don't see answers to these questions:

Apache Arrow data is represented as an opaque byte array in the BatchEvent protobuf message. Is the intent here that it is essentially unprocessable in intermediaries, such as OpenTelemetry Collector and will be simply forwarded as is? If that is not the intent then how can this data be processed? Given that OpenTelemetry Collector is fundamentally built on the current row-oriented data model and all processing capabilities in the Collector work in terms of the row-oriented data model how can the telemetry received in BatchEvent be processed? Do we expect that columnar data will be converted to row-oriented data when it hits the Collector, is processed in row format then converted back to columnar data to exit the Collector? If this is the intent what’s the performance impact of this?

[lquerel]

@tigrannajaryan and @yurishkuro is there anything I can do to complete/clarify this OTEP?

@lquerel I may have missed it but I don't see answers to these questions:

Apache Arrow data is represented as an opaque byte array in the BatchEvent protobuf message. Is the intent here that it is essentially unprocessable in intermediaries, such as OpenTelemetry Collector and will be simply forwarded as is? If that is not the intent then how can this data be processed? Given that OpenTelemetry Collector is fundamentally built on the current row-oriented data model and all processing capabilities in the Collector work in terms of the row-oriented data model how can the telemetry received in BatchEvent be processed? Do we expect that columnar data will be converted to row-oriented data when it hits the Collector, is processed in row format then converted back to columnar data to exit the Collector? If this is the intent what’s the performance impact of this?

@tigrannajaryan I had added yesterday in the section "OpenTelemetry entities to Arrow mapping" some elements to answer your questions and I submitted today some additional clarifications. Please let me know if this is what you expected or if I missed something.

[tigrannajaryan]

I spent a bit more thinking about this proposal.

I think it can be very efficient representation for specialized cases. It is not clear if it can serve as a general-purpose telemetry protocol (the role that OTLP serves today), primarily because it is not clear how intermediary notes like Collector can process data in this format efficiently, while also allowing to define such transformations in a user-friendly way, like current Collector processors do.

I think the best way forward for this proposal would be to:

• Come up with a new protocol name, to make it is clear this is not OTLP, to avoid confusion. Don’t try to encapsulate this in existing OTLP messages, it won’t help existing receivers, they won't be able to interpret it anyway. This is fundamentally a new format. I also would advise against calling this simply OTLP v2, it will likely result in confusion and will prevent more evolutionary changes to current OTLP.
• Get some vendor support who are interested in seeing Otel SDKs emitting this data and contribute exporters for the new protocol to some Otel SDKs.
• Come up with a plan on how the protocol will traverse intermediary nodes (Otel Collector). I still don’t see this in the proposal. The classic way to support a new format is to implement a new receiver and a new exporter for Otel Collector. This will have performance implications that likely are important to know before one can use the format. If one uses the new format instead of OTLP how much you gain in SDK and in the backend and how much you loose in Collector may be an important factor.
• Define when exactly it is beneficial to use this new format/protocol vs OTLP (we can hope that the answer is "always" but likely it will not, so guidelines will be needed).

[carlosalberto]

I second defining this as a new OTLP format (e.g. otlp-arrow), and have the usual prototypes in a few languages.

[anuraaga]

Is my understanding correct that this format's intent isn't RPC but to save for analytics processing? So we'd expect collector to still accept OTLP but save this format to S3?

[lquerel]

@tigrannajaryan, @jmacd thanks for your feedback. I will work on them as soon as possible. I have some other urgent tasks to complete first at F5.

[lquerel]

Is my understanding correct that this format's intent isn't RPC but to save for analytics processing? So we'd expect collector to still accept OTLP but save this format to S3?

@anuraaga, no the intent of this format is to optimize data transfer and in-memory data processing based on a columnar representation. Apache Arrow buffers can be easily converted into Parquet format for storage.

[gramidt]

Is my understanding correct that this format's intent isn't RPC but to save for analytics processing? So we'd expect collector to still accept OTLP but save this format to S3?

@anuraaga - While this format can help for analytics processing, it serves a much greater purpose:

1) Enables multi-variate time-series data. This is critical for systems where metric values are related and wouldn't make sense as individual univariate values (e.g., firewall metrics, multi-axis devices, browser events, etc.). The protocol today only supports univariate (w/ multi-dimensions: attributes, resources) metrics.

2) OTLP as it stands is not performant/cost efficient enough for capturing all data from high-throughput systems (e.g., edge proxies, load-balancers, etc.). This proposal will not only enable capturing data from high-throughput systems, but will significantly reduce the operating cost for processing and receiving telemetry (egress, ingress, compute, etc.).

[jmacd]

OTLP as it stands is not performant/cost efficient enough for capturing all data

Lightstep will corroborate this statement. Customers are looking to Sampling today as an approach to lowering data collection costs for OpenTelemetry trace data. The proposal in this OTEP suggests that users could pay more than an order of magnitude less to collect the same amount of Span data. Where we see this being used to the customer's advantage: OTel collectors would receive ordinary OTLP and then--in an exporter--perform Columnar compression for sending outside their network, i.e., to the vendor across an expensive network link. OTel collector would not need a new pipeline, only a new exporter.

It would be unusual for an exporter to exist and no corresponding receiver or pipeline. Receiving column-compressed metric event data is not very different than receiving Statsd events, they're just more compressed. The receiver will have to perform multivariate-to-univariate metric aggregation in order to inject ordinary OTLP into an OTel collector metrics pipeline. As a user this is not what I'm looking for, but it's something we could build if there is interest.

[gramidt]

It would be unusual for an exporter to exist and no corresponding receiver or pipeline. Receiving column-compressed metric event data is not very different than receiving Statsd events, they're just more compressed. The receiver will have to perform multivariate-to-univariate metric aggregation in order to inject ordinary OTLP into an OTel collector metrics pipeline. As a user this is not what I'm looking for, but it's something we could build if there is interest.

+1 - I believe both a new receiver (multivariate to univariate) and a new straight event pipeline will be needed. There are users ( I am not at liberty to mention their names ) that are looking forward to this enhancement so they can collect and process multivariate time-series. These users currently have multiple environments setup where they use collectors in varying configurations to buffer, filter, and export telemetry.

[carlosalberto]

Where we see this being used to the customer's advantage: OTel collectors would receive ordinary OTLP and then--in an exporter--perform Columnar compression for sending outside their network, i.e., to the vendor across an expensive network link. OTel collector would not need a new pipeline, only a new exporter.

Trying to re-take the conversation - I agree 100% with the previous paragraph. Once we could verify and tune that it works as expected, we could extend this to the rest fo the SDKs later on.

[gramidt]

Where we see this being used to the customer's advantage: OTel collectors would receive ordinary OTLP and then--in an exporter--perform Columnar compression for sending outside their network, i.e., to the vendor across an expensive network link. OTel collector would not need a new pipeline, only a new exporter.

Trying to re-take the conversation - I agree 100% with the previous paragraph. Once we could verify and tune that it works as expected, we could extend this to the rest fo the SDKs later on.

While an exporter is a good start, a pipeline that supports end-to-end multivariate support will also be needed for the Collector. There are fortune 50 companies that use the Collector today and they want to be able to process multivariate time-series using their existing tooling.

[jsuereth]

One thing I haven't seen in this proposal (and I'm curious what the plan for this is going forward) is when you aggregate data in columnar format given the timestamps are shared.

1. Do you ever envision a columnar representations joining Traces Metrics + Logs in one "event"/"batch"?
2. Metrics in OTLP today are arbitrarily aggregated and exported at an interval (vs. actually being multi-variate or sampled at an important event). If we support columnar metrics, should these be synchronous co-exported in place or is the current practice of aggregating (e.g. latency) make sense? Would it make more sense to, e.g. export a Span + related metric points in one columnar export?
3. Logs vs. Events are already called out. Didn't catch the answer to that one.

[lquerel]

@tigrannajaryan @jmacd @jsuereth @carlosalberto @gramidt @yurishkuro Sorry guys for the delay. Between my PTOs and some urgent tasks on my day to day job I have not been able to make any progress in the last few weeks. Fortunately, this will change next week.

[lquerel]

@tigrannajaryan @jmacd @carlosalberto @gramidt @jsuereth @anuraaga @yurishkuro I've tried to summarize in a one diagram the direction I'm taking based on the various feedback. Let me know what you think.

Unless there is a strong disagreement, I'd like to continue to integrate logs and more importantly traces into the benchmark to clarify whether this approach could be global or more focused on the multivariate time-series scenario.

I also think we could phase this project by implementing the multivariate time-series first followed by traces and logs (if the benchmarks confirm my assumption).

Thank you for your patience.

[lquerel]

@jsuereth

One thing I haven't seen in this proposal (and I'm curious what the plan for this is going forward) is when you aggregate data in columnar format given the timestamps are shared.

1. Do you ever envision a columnar representations joining Traces Metrics + Logs in one "event"/"batch"?

No I don't think so. In order to be efficient, batches must be homogeneous. So by design, a batch will be a collection of events sharing the same schema. However, when that makes sense we could represent complex events such as a span event containing related metrics in a single schema (as you suggested in the next section) and consequently build homogeneous batches of such complex events.

2. Metrics in OTLP today are arbitrarily aggregated and exported at an interval (vs. actually being multi-variate or sampled at an important event). If we support columnar metrics, should these be synchronous co-exported in place or is the current practice of aggregating (e.g. latency) make sense? Would it make more sense to, e.g. export a Span + related metric points in one columnar export?

I'm not sure if I understood the questions correctly, so please rephrase if the following answers are not what you expected. Regarding multivariate time-series aggregation, I don't think there is a fundamental difference with the univariate scenario. We should still be able to aggregate multivariate events sharing the same attributes/labels while respecting the rules of aggregation of metrics according to their nature (gauge, counter, ...). Regarding span + related metric, this seems to me a valid scenario of multivariate time-series. We have a set of metrics related to a set of complex attributes representing a span. This representation should avoid data duplication and should also improve processing speed/complexity by eliminating the need to recombine the spans with their corresponding univariate metrics. We might need to separate these complex events from the standard metrics/logs/spans for compatibility reasons. In oltp-arrow metrics, logs and spans can be converted into their counterpart in oltp with sometimes 1:N conversions (e.g. multivariate to n univariate). Things will probably be more challenging for complex events like the one you described. Separating them in a "not oltp compatible" category will make that clear.

[tigrannajaryan]

@lquerel Sorry for delayed response. Thanks for posting the diagram. It makes the intent clear.

I believe what we see here is a need to introduce a new signal type for metrics (metrics-arrow or metrics-columnar, whatever we call it) in the Collector and possibly also for logs and traces similarly. This then necessitates new processors that works with the new signal types. This is potentially a very large amount of work, especially if we are want to be close to feature parity with what processors exist today.

We likely also need to support converting regular data to columnar and back so that existing receivers and exporters for non-OTLP are not left out of the ecosystem. Likely also support for signal type converting in pipelines to make it easier to use together with non-columnar types.

Then we are also looking at implementing columnar exporters in Otel SDKs, likely also other changes necessitated to support columnar data type.

I have a feeling that while technically it is feasible it is unlikely that we as a community are currently able to lift such a major undertaking in the near future.

I do see the benefits and understand how it can lead to significant operational cost savings but at the moment I am very doubtful that this can happen soon simply because I don't see the engineering resources available that can turn this idea into reality.

It is certainly possible to start small, it is not an all-or-nothing endeavour, but again some critical amount of support is needed to get value out of it, which is still non-trivial amount of work and even I don't know if we will be able to lift off.

Sorry if this sounds negative. I like the proposal on the technical merit, but given the limited resources we have at Otel I personally don't see a good way forward right now. I hope I am wrong and there is a way to find engineers to work on this.

[lquerel]

@tigrannajaryan Thank you for your detailed comment. I am aware of the amount of work involved in implementing this OTEP. I am in the process of checking with my company to see if I can invest some of my time in it. I should have an answer by the end of next week. I will most likely start with the multivariate time series implementation as this is the most important part for F5.

I am also creating a trace-based benchmarking tool to allow other companies to test this approach on their own data (OLTP --> OLTP-ARROW, and Columnar SDK --> OLTP-ARROW [this one is not done yet]). @jmacd's team will probably test it soon. See this git repo --> https://github.com/lquerel/oltp-arrow

[carlosalberto]

It is certainly possible to start small, it is not an all-or-nothing endeavour, but again some critical amount of support is needed to get value out of it, which is still non-trivial amount of work and even I don't know if we will be able to lift off.

I agree with the feeling, and hence I would like to state again that working on this feature for Collector export would be a good, small, yet foundational step (the Collector pipeline would stay the same, but additional OTLP-arrow exporters would be created in parallel).

In Lightstep we are interested in this and we may be able to provide some cycles as well.

[lquerel]

@carlosalberto IMO, implementing an OLTP-ARROW exporter will be relatively easy to implement for traces and logs. But I'm still not sure that we can generalize the transformation of a set of related univariate time-series to a multivariate time-series without the collaboration of the time-series producer.

[lquerel]

Just a few words to confirm that I will be able to spend several weeks (this quarter) implementing an OLTP-to-OLTP-Arrow receiver, a basic columnar oriented processor, and an OLTP-Arrow exporter.

[gramidt]

@bogdandrutu @tigrannajaryan @jmacd @jsuereth - We're working on getting further support from a small team within F5 to work on this; however, support from other member organizations would help expedite the progress. Based on the data, the ROI for anyone who receives OTel telemetry would be significant.

I would like to help establish a working group to get this completed. Are any of you able to help recruit resources given the ROI for your organizations?

[tigrannajaryan]

Are any of you able to help recruit resources given the ROI for your organizations?

Unfortunately I don't think I can at the moment.

[lquerel]

@tigrannajaryan @jmacd @carlosalberto @gramidt @jsuereth @anuraaga @yurishkuro Hi all, I'm organizing a meeting next Thursday for a demo of an end-to-end implementation of the OTLP Arrow protocol. Please select your best slots in the following doodle poll and feel free to send this link to anyone interested. Thanks

https://doodle.com/meeting/participate/id/PdRPn3Ea

[tigrannajaryan]

@tigrannajaryan @jmacd @carlosalberto @gramidt @jsuereth @anuraaga @yurishkuro Hi all, I'm organizing a meeting next Thursday for a demo of an end-to-end implementation of the OTLP Arrow protocol. Please select your best slots in the following doodle poll and feel free to send this link to anyone interested. Thanks

https://doodle.com/meeting/participate/id/PdRPn3Ea

Done. It would be useful to add some more slots another day to make it easier to find time that works for all.

[lquerel]

@tigrannajaryan added Friday morning

[lquerel]

According to the poll results, this Thursday 11AM-12PM PST is the best time to do this demo. Please feel free to forward the zoom link below to anyone who might be interested by this presentation/demo. Thanks - Laurent

Zoom meeting link: https://f5networks.zoom.us/j/92928467912?pwd=UFZ5VjhOVG9LeWhtTUNVcFVLUXA1Zz09

[annanay25]

Hi @lquerel, this work looks really interesting.

Is it possible to record the call and publish the recording for those who can't make it?

[lquerel]

Hi Annanay,

I will record the meeting and will publish it.

Thanks

Laurent

On Thu, Dec 16, 2021 at 6:00 AM Annanay Agarwal @.***> wrote:

Hi @lquerel https://github.com/lquerel, this work looks really interesting.

Is it possible to record the call and publish the recording for those who can't make it?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/open-telemetry/oteps/pull/171#issuecomment-995843501, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFAUSVZBKHXVOQ2KPMWALLURHWJHANCNFSM5B47OABQ .

-- Laurent Quérel

[lquerel]

Link to the slides: https://docs.google.com/presentation/d/1u65vz7v4FpNto-t2NGx6D61C9uvk9f4mgKxCMIZKxe8/edit?usp=sharing

[lquerel]

Link to the recording: https://youtu.be/9dGGjREaggY

[tigrannajaryan]

@lquerel thanks for the demo yesterday. A few comments/questions (and I am happy to help you navigate some of these):

Non-string Attributes

Is it possible to have non-string attributes? The schema here shows that attributes are DataType::Utf8. Are other types possible?

Non-primitive Attributes

Is it possible to have non-primitive attribute values, e.g. arrays or maps, including nested?

Non-uniform Attributes

What happens if different records have different types of values for the same attribute name? This is atypical, but may technically happen and is allowed by OTLP today.

Dictionary

Do you store attribute values in the dictionary or only attribute names? If you store the values as well the shared dictionary state can potentially grow unbounded very quickly when high-cardinality data is recorded. Is it possible to limit the dictionary size somehow? Is it possible to have dictionary-less mode of operation for use-cases when memory usage from shared state is a concern?

Can you seed the dictionary with Otel semantic conventions, so that these are never needed to be transferred on the wire?

gRPC Streams

gRPC streams are not load balancer friendly. They use persistent connections and load balancers don't rebalance them. Over time this can result in a significant dis-balance in the target load. We have observed this with OpenCensus protocol which used gRPC streams.

One possible approach is to forcedly reconnect periodically.

Benchmarks

It would be useful to do benchmarks for very small batch sizes and also with real-world data, such as for example data from Collector's hostmetricsreceiver or from Otel Helm Chart from Kubernetes.

Naming

If the new protocol is not going to be interoperable with existing versions of OTLP receivers then I don't think we can call it OTLP. It will be misleading and source of confusion. If there is a way to make old and new versions of senders and receivers interoperate seamlessly then it is ok to use the name. You may want to look into having the Arrow stream as an optionally attempted mode with a fallback to the old mode if the receiver doesn't support it.

Schema

The word "Schema" already has a meaning in OpenTelemetry and means something very different. It may be useful to disambiguate this in your proposal to avoid confusion.

Collector

I touched this in the meeting, but want to emphasize one more time. There is a very large ecosystem around the current Collector already. It can be very useful to build on top of this ecosystem instead of trying to build a new one. There is over 120 components in the Collector and re-implementing these would be a huge undertaking (even if not all processors are needed). There is a large number current Collector contributors who don't necessarily know Rust. It is going to be very difficult to generate a similar momentum around a new Collector.

I would highly advise to look into ways to make the new protocol work with current Collector. Perhaps one first interesting step would be to implement the protocol as Collector receiver and exporter. If it demonstrates much lower wire size / network cost that is sufficient to start the ball rolling, even if it means there is performance impact because of the need to convert to/from pdata internal memory representation.

[jpkrohling]

I just finished watching the presentation, thank you for delivering and recording it. I found it highly interesting and I understand the choice of Rust given the ecosystem around Apache Arrow. But I share the same concerns as @tigrannajaryan: we cannot expect our current collector ecosystem to be rewritten in Rust, especially given that we are late in delivering a v1 already.

That all said, I am excited about a columnar-friendly alternative to OTLP, potentially with a different name as @tigrannajaryan mentioned earlier.

[lquerel]

@tigrannajaryan Thank you for your very detailed feedback.

In general, I understand the need to leverage this large ecosystem and will definitively give your last suggestion some serious thought. I wasn't aware of the restriction regarding integration with C APIs. This obviously complicates my initial plan and makes integration with libraries like DataFusion also problematic. I hope this vacation season will be a good opportunity to think about my options.

Answers to your questions:

• Non-string attributes: There is no restriction on the type of column attributes.
• Non-primitive attributes: Arrays and nested structs are supported by Arrow but are currently not supported by DataFusion. So at the protocol level it's ok (even if they are more complicated to handle) but some complex processors like SQL will not be able to process them.
• Non-uniform attributes: batches are grouped by schema signature and the signature take into account the name and type of the columns defined in the Arrow Schema.
• Dictionary: dictionaries are optional and can be used on any columns containing Utf8 values. If a dictionary is defined for an attribute column then the values will be stored in the dictionary and the column will contain the index on the dictionary entries. It's better to think about dictionaries as an optimization mechanism for low-medium cardinality attributes. This could be defined (e.g. via annotation) by some instrumentation of the client library.
• gRPC streams: This is a problem I'm aware of. Several Google APIs (e.g. PubSub, BigQuery Write Storage) rely on gRPC streams for performance reason and use the type of mitigation you mentioned in addition to encouraging clients to open multiple streams in parallel to distribute the load over multiple targets.
• Benchmark: Yes this is something I will definitively do.
• Naming: I'd like to better understand your proposal here (i.e. You may want to look into having the Arrow stream...).
• Schema: Ok for now I will use the term Arrow Schema but I'm open to better suggestion.
• Collector: I understand your concern and need to think more about it.

[tigrannajaryan]

• Naming: I'd like to better understand your proposal here (i.e. You may want to look into having the Arrow stream...).

@lquerel current OTLP version is v0.11.0. Let's assume we introduce OTLP v0.12.0 which adds support for Arrow streams. Can we make senders of v0.12.0 interoperable with receives of v0.11.0? If we can make it happen then I think we can safely call this protocol OTLP, otherwise it would be confusing. I think such interoperability is possible to achieve:

• Sender v0.12.0 attempts to send data using new Arrow gRPC stream.
• Receiver v0.11.0 will likely respond with an error to the gRPC stream establishment request (need to verify this)
• Sender v0.12.0 receives the error response and if it is able to correctly identify that the error is caused by Receiver not supporting Arrow stream, the sender falls back to non-Arrow mode of operation, i.e. the v0.11.0 mode of operation.
• Sender v0.12.0 continues functioning as if it was a sender v0.11.0.

[yurishkuro]

@lquerel thanks for the demo. As I mentioned during the meeting, I am very interested in seeing how this could work from the SDK side. The backend pipeline you demoed has a strong dependency on fixed schemas for different data streams, yet "schemas" as such play no role in the OpenTelemetry APIs today, all data recording (traces and metrics) is done via open-ended Map<String, Any>. Even though, in practice, any given service would normally have only a small number of "shapes" for the spans (and larger but still fixed number of shapes for metrics), there is no mechanism for distinguishing which event is recorded with which shape. One naive way would be for the exporter to group the events by their schemas, which could be pretty expensive computationally (e.g. sorting attributes and computing a hash).

Or did you envision Arrow encoding only in the backend pipelines and not from the SDKs?

[lquerel]

@tigrannajaryan yes, I think it is perfectly achievable and I will check its feasibility soon. I like your approach because it fits perfectly into the current ecosystem while allowing gateways that natively support end-to-end arrow streams to target bandwidth optimized scenarios (see Lightstep use cases described by @jmacd ).

@yurishkuro I think both scenarios will coexist (at least at first).

• Scenario 1: the current SDK sends the data to a gateway supporting arrow streams allowing to apply complex transformations on the data and optimizing the bandwidth for the compatible backends. In this scenario, the protocol between the SDK and the collector is the same as today.
• Scenario 2: a future evolution of the SDK allows to reify this stream concept up to the client level to report without overhead metrics, traces that have the same shape.

The choice to use this option in the SDK being left to the user.

Concerning the important variability of the shapes for the traces, I would like to study a little more closely a concrete case. Would you have a simple and reproducible scenario? Do you think that the Synthetic Load Generator Utiliy described in the performance page is a good candidate to produce this type of trace?

[tigrannajaryan]

IIRC, Synthetic Load Generator Utiliy produces fairly static data, so not a good example of variability.

[lquerel]

@tigrannajaryan I confirm that we can detect the presence of a new stream oriented service and fallback on the existing services if not implemented by the receiver.

My plan is:

1. Support a configuration with a Go Collector sending metrics/logs/traces to the Rust Collector in order to complete the transformation logic between the current version of the protocol and the new one.
2. Support a configuration with a Rust Collector sending metrics/logs/traces to the Go Collector to support the reverse transformation.
3. Check traces scenarios based on @yurishkuro feedback.
4. Update the OTEP according to these modifications
5. Enter in the review process, refine OTEP and obtain approval.
6. Update the Go OTLP receiver to expose the stream oriented service.
7. Update the Go OTLP exporter to transform and send stream oriented events.

The last two steps could be done with the help of the community.

Looking forward to your feedback.

[jgehrcke]

Hello! Great to see this here. I have not followed the conversation in all detail but skimmed over the current Motivation and Explanation sections.

I would like to add a motivational aspect. Encapsulated binary Arrow messages have the advantage that their layout can be known / read prior to reading the bulk of the data. That is, a rate-limiting decision can be made in the receiving end after reading only a few bytes from a (TCP) connection. That could be a huge win for larger-scale environments compared to the current/legacy Prometheus (protobuf) way of communicating larger chunks of metric/log data (here, the receiving end typically reads and deserializes the complete protobuf message before it can make a rate-limiting decision).

Happy to elaborate/discuss.

[lquerel]

@jgehrcke I will add this in the motivation section. In the last iteration of the arrow-based protocol, I separated the declaration of the Arrow Schema from the RecordBatch, so it's even slightly better for this use case. The schema is only sent one time per stream and delta dictionaries is also supported.

[mdisibio]

Hi, this proposal is very interesting and I have been following along for some time. Has there been any recent development? We are pursuing columnar trace storage in Grafana Tempo and would like to converge on something compatible if possible.

In particular the open questions around attribute shape and how to translate the map<string,any> to columnar effectively.

Arrow integration has many benefits for processing, but since we are focused on the disk format, it is not immediately required. Happy to discuss details and thinking if there is interest.

[lquerel]

Hi @mdisibio, Short answer yes I'm still working on this proposal and its implementation (Rust and Go), but not at the speed I'd like.

I worked on the representation of attributes which are in fact a kind of hierarchical representation. I ended up with the following Arrow representation:

• KeyValueList are mapped to Arrow Struct. The rational being that for a specific type of metrics, logs, or traces the keys of the KeyValueList are most of the time the same. For a specific batch the struct fields are the merging of all the keys, so even when we have a slight difference between the attributes it's not a big deal.
• ArrayValue are mapped to Arrow List
• All the other mapping are straightforward.

I also changed a bit the way the OTLP entities are represented in Arrow RecordBatch to get rid of some duplications that make sense in the context of a telemetry protocol (not necessarily the best option for a disk format).

• Metrics entities are represented with 3 Arrow RecordBatch (and their corresponding dictionaries): Resources, InstrumentationLibraries, and Metrics Arrow record batches. The one to many relationship between Resource and Metrics is concretized by a numerical resource_id. I use a similar approach for the relationship between InstrumentationLibrary and Metrics.
• Logs follow a similar pattern (see below).
• Spans follow a similar pattern and add two additional RecordBatch: event and link with a relationship many to one.

I am interested in discussing this with you in more detail. We definitely have everything to gain by finding the best tradeoff between disk and wire format to optimize the overall performance. Please contact me via email: l.querel@f5.com or laurent.querel@gmail.com.

[lquerel]

Hi everyone, This new update is a much more complete version of the specification including

• the new service EventService and the corresponding protocol description,
• mapping OTLP -> OTLP Arrow entities, AnyValue -> Arrow data types, ...
• a set of benchmarks,
• parameter tuning and design optimization analysis.

New version of OTEP 0156 here.

The initial POC (Rust implementation) served as a validation for the specifications and for the benchmarks. However, a Go library converting OTLP entities into OTLP Arrow entities is in progress. This library should allow the community to integrate the receiver and exporter described in the specification.

@tigrannajaryan , @jmacd , @yurishkuro , @jsuereth, @gramidt , @carlosalberto , @jpkrohling , @pirgeo , @mdisibio, @jgehrcke , @annanay25 , @anuraaga - I am waiting for your feedback. Thanks in advance.

[tigrannajaryan]

@lquerel thanks for updating the OTEP. I am travelling and mostly AFK, but will take a look next week. If you have any Go code that implements this it would be also great to see it.

[lquerel]

@tigrannajaryan "If you have any Go code that implements this it would be also great to see it."

This is definitely the plan. I hope to make this public this summer.