search

NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
50 stars 37 forks source link

Qualification tool should print Kryo related recommendations #1204

Closed tgravescs closed 2 months ago

tgravescs commented 2 months ago

fixes https://github.com/NVIDIA/spark-rapids-tools/issues/1106

Qualification tool should print Kryo related recommendations

The Qualification tool will recommend spark.kryo.registrator=com.nvidia.spark.rapids.GpuKryoRegistrator for Kryo if the CPU eventlog has set spark.serializer=org.apache.spark.serializer.KryoSerializer.

Note this is built upon pr so marking as draft until https://github.com/NVIDIA/spark-rapids-tools/pull/1188 is merged. And the diff here is way wrong, essentially the change here is the new function - recommendKyroSerializerSetting

tgravescs commented 2 months ago

1188 merged so this was upmerged and should be ready

tgravescs commented 2 months ago

@parthosa please take another look

amahussein commented 2 months ago

Thanks @tgravescs Squashing the commits through Github will leave the 93 commits messages. Please make sure to clear up those lines before merging the PR.