search

apache / incubator-xtable

Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines.
https://xtable.apache.org/
Apache License 2.0
922 stars 148 forks source link

Open Telemetry Integration #37

Open the-other-tim-brown opened 1 year ago

the-other-tim-brown commented 1 year ago

Instrument the project with open telemetry or something similar to get a better understanding of the performance characteristics of Onetable

taher-cldcvr commented 11 months ago

@the-other-tim-brown I would love to take a look at this. Can you give me some starting points?

the-other-tim-brown commented 11 months ago

@ashvina correct me if I'm wrong on any of this.

I think the general idea was to integrate with https://opentelemetry.io/ so we can export information about how long the conversion is taking to allow users to better understand the overhead. With OTel specifically, we can even introduce spans for sections of the code to allow finer grain views instead of a simple metric reporting total conversion time. This allows us to point to specific parts of the code taking more time than others.

ashvina commented 11 months ago

Hi @taher-cldcvr Good timing. This feature seems to be getting some traction. Our main goal is to generate valuable telemetry. As Tim pointed out, opentelemetry is a good option. Do you have experience with OT? It would be great to start a design proposal on this and discuss. Thoughts?

taher-cldcvr commented 11 months ago

@ashvina I do not have experience with open telemetry, however, on exploration this seems more like a mix of sentry and metrics reporting, which I should be able to start a design discussion. I will put up a doc around this and get a community opinion on this.