NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
45 stars 34 forks source link

Improve CSP qualification and profiling tool validation #835

Open tgravescs opened 2 years ago

tgravescs commented 2 years ago

Is your feature request related to a problem? Please describe. The validation we are doing for running profiling tool on CSPs is very basic, just checking that the file exists for now because we are running it on the integration test event logs and depending on the CSP they are run differently and some have failure cases and such. Come up with better validation or perhaps different test cases. The main thing we want to validate is just that is run on those CSPs Spark version and event logs they generated, so perhaps if we can look at a very specific event log for a test we know generates certain things.

tgravescs commented 2 years ago

Note for the qualification tool we shut off csp validation because it no longer runs on GPU longs. We need to come up with different option for this.