[BUG] Estimate endTime for non-terminated SQLs in Profiling tool

NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS

Apache License 2.0

48 stars 35 forks source link

[BUG] Estimate endTime for non-terminated SQLs in Profiling tool #1170

Closed amahussein closed 1 month ago

amahussein commented 1 month ago

Is your feature request related to a problem? Please describe.

The Profiling tool shows empty duration for Non-terminated SQLs. SHS treats those SQLs as "running". The tools should estimate the endTime to calculate a duration.

We need to decide on the strategy to estimate the endTime:

is it the max EndTime of last successful task if any?
the max endTime of a task if any (either successful/failed)?

amahussein commented 1 month ago

Waiting to hear back more specification about what is the desired duration.

amahussein commented 1 month ago

closing this for now as further information is required