Open rickFanta opened 1 month ago
Yes, implementing such an exporter was part of our plan. Storing the Arrow records as they are would be easy, but in my opinion, it wouldn’t be ideal or optimal. The current Arrow schema used in the protocol has been optimized for transport, taking into account the need to regularly close streams (and therefore reset states on the receiver side) to make the protocol more load-balancer-friendly. Some changes or transformations need to be applied first to optimize records for long-term storage. Unfortunately, I don’t have an ETA to provide at this time.
Thanks for the reply. There's strong perceived value here in otlp-flavor data stored as arrow/feather to allow quick tactical queries via DuckDB and similar, etc.
Any thoughts you have on how to enact your best solution, or do a mostly durable tactical one, would be very much appreciated, if you have the cycles. Happy to try to help towards either/both.
Might it be possible to add a FileExporter to this to allow export to arrow or feather (or parquet)???
Or is there an obvious way to do this that I'm missing?