NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
44 stars 34 forks source link

[BUG] Qualification tool should use system properties from eventlog for Auto tuner #1024

Closed tgravescs closed 1 month ago

tgravescs commented 1 month ago

Describe the bug The qualification tool, specifically the auto tuner part should use the properties from the event log if either the cluster properties aren't passed in or are incomplete.

amahussein commented 1 month ago

The AutoTuner reads the properties from both:

If the property is not defined by the eventlog, it falls back to read it from the worker-info

  def getPropertyValue(key: String): Option[String] = {
    val fromProfile = appInfoProvider.getProperty(key)
    // If the value is not found above, fallback to cluster properties
    fromProfile.orElse(Option(clusterProps.softwareProperties.get(key)))
  }

Closing the issue as invalid for now.