With https://github.com/NVIDIA/spark-rapids-tools/pull/686 , the incomplete evetnlogs would throw JsonEOFException. However, there is an eventlog sample that is incomplete and Spark would instead throw a JsonParseException causing the entire analysis to be skipped.
Upon investigation, I found that there are some cases that causes different failure.
To reproduce, I truncated TaskEnd event line and triggered Profiler tool.
As a result, the profiler shows that there is a JSONParseException and set teh status to UNKNOWN. Instead, we can try to handle that corner case in order to avoid skipping all incompletefiles.
Describe the bug
With https://github.com/NVIDIA/spark-rapids-tools/pull/686 , the incomplete evetnlogs would throw
JsonEOFException. However, there is an eventlog sample that is incomplete and Spark would instead throw a JsonParseException causing the entire analysis to be skipped.Upon investigation, I found that there are some cases that causes different failure.
To reproduce, I truncated TaskEnd event line and triggered Profiler tool.
As a result, the profiler shows that there is a JSONParseException and set teh status to UNKNOWN. Instead, we can try to handle that corner case in order to avoid skipping all incompletefiles.