The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
Is your feature request related to a problem? Please describe.
It's hard to dig into the details while troubleshooting a query. In our use case, queries from clients are sent to Starrocks using the JDBC protocol, and Datadog is the observability platform for log/metric collection, etc.
It lacks some core features when looking into the details of the query processing:
there's no trace for SQL queries to show the processing steps, communication between FE/BE, etc;
not able to find all the log events based on query_id;
Describe the solution you'd like
With OpenTelemetry, we build a platform-independent trace record for each SQL request to expose the processing information, including spans and log events, within/between FE and BE components. It includes several improvements:
build trace record for each SQL request;
support trace context propagation from client;
support trace context from FE to BE;
attach trace context to log events;
Describe alternatives you've considered
Additional context
We run a PoC(proof-of-concept) to achieve some of the requirements, the solution is implemented as below:
trace context is added as a comment to the SQL;
In FE, it starts a trace record for each query, with the parentContext extracted from SQL comments when provided;
trace context is added to log events as tags with MDC, BE is not covered in the PoC;
mingmxu
commented
2 months ago
Feature request
Is your feature request related to a problem? Please describe. It's hard to dig into the details while troubleshooting a query. In our use case, queries from clients are sent to Starrocks using the JDBC protocol, and Datadog is the observability platform for log/metric collection, etc.
It lacks some core features when looking into the details of the query processing:
Describe the solution you'd like With OpenTelemetry, we build a platform-independent trace record for each SQL request to expose the processing information, including spans and log events, within/between FE and BE components. It includes several improvements:
Describe alternatives you've considered
Additional context We run a PoC(proof-of-concept) to achieve some of the requirements, the solution is implemented as below: