NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
49 stars 36 forks source link

[BUG] The raw_metrics CSV file miss App endTime and appDuration #1091

Closed amahussein closed 3 months ago

amahussein commented 3 months ago

Describe the bug After the refactor #1000 , it looks like that missing to call appInfo.aggregateInfo() resulted in generating Profiler output before the appEndTime is calculated. As a result, the raw_metrics inside Qualification output-folder will contain None when the eventlog is prematurely terminated (i.e., case of Databricks)