search

linkedin / dr-elephant

Dr. Elephant is a job and flow-level performance monitoring and tuning tool for Apache Hadoop and Apache Spark
Apache License 2.0
1.35k stars 859 forks source link

Relaxing Buffer Fraction for Spark Driver Memory #673

Closed saniyanajeeb closed 4 years ago

saniyanajeeb commented 4 years ago

Increasing buffer fraction in Spark Driver Memory Heuristic.

pralabhkumar commented 4 years ago

@saniyanajeeb please put the test case to check the functinality

saniyanajeeb commented 4 years ago

Tested the changes on Dr Elephant test machine for spark jobs running on prod cluster: Few examples after increasing the buffer fraction to 0.5:

  1. For one application, The spark.driver.memory is now being suggested as 4 GB. With the older value of buffer fraction (0.2), this would have been 2.74 GB rounded off to 3 GB.

    Screenshot 2020-04-02 at 7 54 36 PM
    1. For another application, The spark.driver.memory is now being suggested as 4 GB. With the older value of buffer fraction (0.2), this would have been 2.83 GB rounded off to 3 GB. Screenshot 2020-04-02 at 8 03 04 PM

@pralabhkumar

pralabhkumar commented 4 years ago

LGTM