search

open-telemetry / otel-arrow

Protocol and libraries for sending and receiving OpenTelemetry data using Apache Arrow
Apache License 2.0
83 stars 15 forks source link

Add a FileExporter to allow export to arrow or feather (or parquet)??? #250

Open rickFanta opened 1 month ago

rickFanta commented 1 month ago

Might it be possible to add a FileExporter to this to allow export to arrow or feather (or parquet)???

Or is there an obvious way to do this that I'm missing?

lquerel commented 1 month ago

Yes, implementing such an exporter was part of our plan. Storing the Arrow records as they are would be easy, but in my opinion, it wouldn’t be ideal or optimal. The current Arrow schema used in the protocol has been optimized for transport, taking into account the need to regularly close streams (and therefore reset states on the receiver side) to make the protocol more load-balancer-friendly. Some changes or transformations need to be applied first to optimize records for long-term storage. Unfortunately, I don’t have an ETA to provide at this time.

rickFanta commented 1 month ago

Thanks for the reply. There's strong perceived value here in otlp-flavor data stored as arrow/feather to allow quick tactical queries via DuckDB and similar, etc.

Any thoughts you have on how to enact your best solution, or do a mostly durable tactical one, would be very much appreciated, if you have the cycles. Happy to try to help towards either/both.