search

PengNi / basemods_spark

0 stars 0 forks source link

take care of configuration of Spark #3

Open PengNi opened 6 years ago

PengNi commented 6 years ago

Spark Properties

spark.executor.memory spark.cores.max spark.task.cpus

spark.storage.memoryFraction spark.memory.fraction spark.memory.storageFraction

spark_local_dirs (in spark-env.sh)

spark.memory.offHeap.enabled spark.memory.offHeap.size

for spark on yarn yarn.nodemanager.local-dirs (for yarn) yarn.nodemanager.vmem-check-enabled (set it to false when necessary, ref: github) spark.yarn.executor.memoryOverhead