search

open-telemetry / opentelemetry-specification

Specifications for OpenTelemetry
https://opentelemetry.io
Apache License 2.0
3.75k stars 889 forks source link

OpenTelemetry Arrow Project #3316

Closed jmacd closed 1 year ago

jmacd commented 1 year ago

Description

This OpenTelemetry project will coordinate the introduction of Apache Arrow features and technologies into OpenTelemetry. This project has been scoped in two phases, described in the Community Donation and the original OTEP. These phases help understand project scope:

  1. Introduce an Arrow "Bridge" to allow one OTel Collector to send data compressed in Apache Arrow between one OTel Collector and another. This includes a Golang library to convert standard Collector pdata objects into Arrow data frames and the reverse translation.
  2. Propose further extensions to OTel, including Arrow-first SDK code paths that support direct Arrow export and an Arrow-first pipeline that supports data manipulation directly in Arrow format.

Project Board

@jmacd is the assigned Technical Committee member for this project. Once approved, this project board will be populated with tasks and milestones. Presently we have been tracking medium and small issues in two repositories https://github.com/open-telemetry/experimental-arrow-collector/issues and https://github.com/f5/otel-arrow-adapter/issues.

Once created, please link to the project board here.

Deliverables

For our first phase, which has been heavily documented and prototyped, the deliverables are:

  1. The "OTel-Arrow-Adapter" library which converts to and from Arrow and OTC's pdata.
  2. The OTel-Arrow Receiver component, to be placed in OTel-Collector-Contrib.
  3. The OTel-Arrow Exporter component, to be placed in OTel-Collector-Contrib.

We expect to plan and design second-phase deliverables after this project is accepted and after phase 1 deliverables are starting to see use. The two vendors that have sponsored the prototype deliverables, F5 and Lightstep, expect to continue contributing and welcome the community's input as we plan phase 2 deliverables.

Staffing / Help Wanted

We have sufficient staffing for our phase 1 deliverable. Phase 2 deliverables vary in scope and complexity and are likely areas where input will be welcome. If you are interested in direct SDK support for Arrow or direct Arrow-based telemetry pipelines, please join us!

Required staffing

Project lead: @lquerel Lead TC member: @jmacd Contributing engineers: @lquerel and @jmacd

We need to gather one OTel-Collector maintainer and one more TC member here.

Meeting Times

We have been meeting Tuesday's at 9am PST/PDT.

Timeline

Phase 1 deliverables will be ready for production use in Q2, 2023. Phase 2 deliverables are T.B.D.

Labels

T.B.D.

Linked Issues and PRs

OTEP: https://github.com/open-telemetry/oteps/pull/171 Prototype repositories: https://github.com/open-telemetry/experimental-arrow-collector and https://github.com/f5/otel-arrow-adapter

reyang commented 1 year ago

We need to gather one OTel-Collector maintainer and one more TC member here.

I can be the "one more TC member here".

tigrannajaryan commented 1 year ago

Project is up and running. Closing the issue.