1000 introduced a new interface design for calculating ProfileResults types. This enabled qualification tools to generate these results as part of the raw_metrics folder.
This PR refactors the calculation of the remaining ProfileResult types to follow the same design pattern. These files are needed by the estimation model. This is a step toward using only the qualification tool for the estimation model.
Files generated by Qual Tool
application_information.csv
application_log_path_mapping.csv
data_source_information.csv
Properties files:
spark_properties.csv,
system_properties.csv
spark_rapids_parameters_set_explicitly.csv
Changes:
Java/Core
Refactored CollectInformation: Moved methods into the new framework ViewableTrait[R <: ProfileResult].
Created traits and objects for each ProfileResult case.
Testing:
Manually tested JAR on event logs to verify Qual Tool generates these files.
Profiling unit tests are unaffected as this is an internal refactor.
We do not have unit tests for the raw_metrics generated by Qual Tool. We can create followup issue for this.
Fixes #1041.
1000 introduced a new interface design for calculating
ProfileResults
types. This enabled qualification tools to generate these results as part of the raw_metrics folder.This PR refactors the calculation of the remaining
ProfileResult
types to follow the same design pattern. These files are needed by the estimation model. This is a step toward using only the qualification tool for the estimation model.Files generated by Qual Tool
application_information.csv
application_log_path_mapping.csv
data_source_information.csv
spark_properties.csv
,system_properties.csv
spark_rapids_parameters_set_explicitly.csv
Changes:
Java/Core
CollectInformation
: Moved methods into the new frameworkViewableTrait[R <: ProfileResult]
.ProfileResult
case.Testing:
raw_metrics
generated by Qual Tool. We can create followup issue for this.