Signed-off-by: Ahmed Hussein (amahussein) a@ahussein.me
Fixes #1042
Running the tool on a legacy spark eventlog (2.x) caused problems because the SQL metric names are not the same
Changes
This PR adds a map between <legacy-metric-names, new-metric-names>
The map is looked-up in the following parts:
During the construction of an AccumulableInfo: this guarantees that all metrics created in a TaskEnd/Stage are mapped to the valid names
During the contruction of the SQLplanGraph: this guarantees that node metrics are mapped to the most recent labels
I added an extra match-pattern to avoid this failure from happening in the future. When the mtric-name is missing, there will be a warning message that it is not handled correctly.
The process to do the fix was painful because I had to look into all spark versions to spot any metric-name change. t is possible that I did not catch them all, but at least we can append more mapping in the future.
I had to change UT because the changes actually fixed a hidden bug in our code.
Signed-off-by: Ahmed Hussein (amahussein) a@ahussein.me
Fixes #1042
Running the tool on a legacy spark eventlog (2.x) caused problems because the SQL metric names are not the same
Changes
This PR adds a map between <legacy-metric-names, new-metric-names>
I had to change UT because the changes actually fixed a hidden bug in our code.