imatiach-msft
commented
2 years ago @JWenBin I would recommend to try to avoid numBatches if possible. All it does is split the dataset into multiple batches, train the model on the first batch, then retrain it on the second batch, etc. It will be much less likely to run out of memory, as you increase the number of batches, but it will also likely decrease model accuracy and increase training time. It looks like you have 26320507 rows - how many columns are there? What is your cluster size - the number of nodes, CPUs per machine and RAM per machine? Based on this submission command you posted: --num-executors 1 --executor-memory 50G --executor-cores 47 It looks like you are using a single executor with 50GB RAM allocated to it and 47 cores to run distributed training: https://community.cloudera.com/t5/Support-Questions/Spark-num-executors-setting/td-p/149434 The --num-executors is the total number of executors, did you intend to run all training on just a single machine/executor?
shuDaoNan9
kyoungrok0517
If I do not set numBatches, there will be ‘NegativeArraySizeException’ or ‘OOM’ during trainning big dataset (about 26320507 rows), and the cpu utilization will be less than 90%. But if I set numBatches, it will take more time to train the model, although the cpu utilization will be about 97% during each batch.
submit: spark-submit --master yarn --driver-cores 2 --driver-memory 5G --num-executors 1 --executor-memory 50G --executor-cores 47 --conf spark.driver.maxResultSize=3G --conf spark.yarn.heterogeneousExecutors.enabled=false --conf spark.dynamicAllocation.enabled=false --jars ......
code:
NegativeArraySizeException log:
Thank you!
AB#1884827