Configuring cluster to increase the max number of files that can be open (needed by Spark during large shuffles)

[jbherman]

i saw you encountered this problem during big shuffles. here is how i fixed it. hope it helps. Note: this is on amazon linux.

1) copy os's sysctl.conf 2) append "fs.file-max = 100000" to sysctl.conf 3) copy-file bigCluster /Users/jason/projects-misc/sysctl.conf /home/ec2-user/sysctl.conf 4) run-command bigCluster "sudo cp /home/ec2-user/sysctl.conf /etc/sysctl.conf" 5) run-command bigCluster "sudo shutdown -r now" 6) cat < limits.conf

• soft nproc 65535
• hard nproc 65535
• soft nofile 65535
• hard no file 65535 EOF 7) copy-file bigCluster /Users/jason/projects-misc/limits.conf /home/ec2-user/limits.conf 8)run-command bigCluster "sudo cp /home/ec2-user/limits.conf /etc/security/limits.conf" 9)run-command bigCluster "sudo shutdown -r now"

thanks for your great work. -jason

[nchammas]

Hi Jason, and thank you for the kind words!

Is this in reference to this comment?

[jbherman]

yes it's a reference to that comment