nchammas / flintrock

A command-line tool for launching Apache Spark clusters.
Apache License 2.0
637 stars 116 forks source link

ValueError: current limit exceeds maximum limit #216

Closed xiandong79 closed 6 years ago

xiandong79 commented 6 years ago
  1. flintrock --version
    Traceback (most recent call last):
    File "/usr/local/bin/flintrock", line 11, in <module>
    sys.exit(main())
    File "/usr/local/lib/python3.6/site-packages/flintrock/flintrock.py", line 1125, in main
    set_open_files_limit(4096)
    File "/usr/local/lib/python3.6/site-packages/flintrock/flintrock.py", line 1105, in set_open_files_limit
    (min(desired_limit, hard_limit), hard_limit))
    ValueError: current limit exceeds maximum limit

I have to use sudo to fix the problem.

nchammas commented 6 years ago

Hmm, you should not need sudo. What version of macOS are you running and what does ulimit -aH return?

xiandong79 commented 6 years ago
  1. macOS Sierra version 10.12.6

  2. ➜  ~ ulimit -aH
    -t: cpu time (seconds)              unlimited
    -f: file size (blocks)              unlimited
    -d: data seg size (kbytes)          unlimited
    -s: stack size (kbytes)             65532
    -c: core file size (blocks)         unlimited
    -v: address space (kbytes)          unlimited
    -l: locked-in-memory size (kbytes)  unlimited
    -u: processes                       1064
    -n: file descriptors                10240
nchammas commented 6 years ago

That's strange. Your hard limit for file descriptors is 10240, and Flintrock is requesting less than that (4096) so it should work.

When you get that error, what does ulimit -n show?

xiandong79 commented 6 years ago
~ ulimit -n
1024

By the way, the launching speed is amazing !!! You really did something!

nchammas commented 6 years ago

Thank you. 😄

What do you get if you run ulimit -n 4096 followed by ulimit -n?

xiandong79 commented 6 years ago
~  ulimit -n
1024
  ~  ulimit -n 4096
ulimit: setrlimit failed: invalid argument
xiandong79 commented 6 years ago
~ ulimit -n 8192
➜  ~ ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       709
-n: file descriptors                8192
➜  ~ ulimit -n
8192
xiandong79 commented 6 years ago

It works well now!

nchammas commented 6 years ago

I'm confused how the second attempt worked (8192) but the first didn't (4096).

You may have to adjust some kernel limits to let Flintrock modify the file descriptor limit without sudo. This discussion should help guide you.

nchammas commented 6 years ago

Glad to hear it works now!