spark performance

[GunSik2]

검토사항

• s3 committer
• shuffle performance with volume : docker filesystem option - hostpath, tmpfs
• remote storage for persisting shuffle data (SPARK-25299)
• After optimisations(read-ahead,speculation) very high resource utilisation and significant improvement of execution time
• Workload scaled near-linear with increasing number of executors

성능분석도구

• NVIDIA Nsight Systems

참고자료

• https://adatis.co.uk/performance-enhancements-in-delta-lake/
• https://databricks.com/session_na20/running-apache-spark-on-kubernetes-best-practices-and-pitfalls
• https://indico.cern.ch/event/754246/contributions/3124983/attachments/1708398/2753279/Spark-Kubernetes-Training-2018.pdf
• Spark Streaming and HDFS ETL with Kubernetes
• spark-performance-benchmarks 참고도표