resource limit problem when running locally on OSX

[porterjamesj]

This is a bit of a weird one and as far as I can tell is either an OSX or a Python bug, not yours. When trying to run locally on OSX, I get the following:

with cluster_view("test","test",1,extra_params={"run_local":True}) as view:
    view.map_sync(lambda x: x+1,[1,2,3])

prints

[ProfileCreate] Generating default config file: u'/Users/james/.ipython/profile_6e3e178c-cb04-11e3-87aa-7c6d628c4150/ipython_config.py'
[ProfileCreate] Generating default config file: u'/Users/james/.ipython/profile_6e3e178c-cb04-11e3-87aa-7c6d628c4150/ipython_nbconvert_config.py'
[ProfileCreate] Generating default config file: u'/Users/james/.ipython/profile_6e3e178c-cb04-11e3-87aa-7c6d628c4150/ipcontroller_config.py'
[ProfileCreate] Generating default config file: u'/Users/james/.ipython/profile_6e3e178c-cb04-11e3-87aa-7c6d628c4150/ipengine_config.py'
[ProfileCreate] Generating default config file: u'/Users/james/.ipython/profile_6e3e178c-cb04-11e3-87aa-7c6d628c4150/ipcluster_config.py'
[ProfileCreate] Generating default config file: u'/Users/james/.ipython/profile_6e3e178c-cb04-11e3-87aa-7c6d628c4150/iplogger_config.py'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ValueError: current limit exceeds maximum limit
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ValueError: current limit exceeds maximum limit

I dug into it a bit, on my machine resource.getrlimit(resource.RLIMIT_NOFILE) gives (2560, 9223372036854775807). That later value is RLIMIT_INF, since OSX "doesn't enforce limits" on the number of open files. The scare quotes are because when I actually try to set the limits as cluster_helper does:

In [3]: resource.setrlimit(resource.RLIMIT_NOFILE, (50000, 9223372036854775807))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-868cbfc99723> in <module>()
----> 1 resource.setrlimit(resource.RLIMIT_NOFILE, (50000, 9223372036854775807))

ValueError: current limit exceeds maximum limit

There apparently is a limit, it just isn't advertised correctly. I'm working around this for now by chaning cluster.cluster_cmd_argv to lower target_procs. It isn't really a production issue since you're it's unusual to be running analysis on OSX servers :). It is fairly annoying when testing though , so I figured I'd let you know in case you want to add some sort of hack around it to the package.

cheers!

[porterjamesj]

more info; apparently OSX has a secret OPEN_LIMIT: http://www.engardelinux.org/modules/index/list_archives.cgi?list=postfix-devel&page=0025.html&month=2013-03

[porterjamesj]

classic apple, I'm not sure python even has an API to work around this.

[chapmanb]

James; Thanks for the heads up and sorry about the issue. It's annoying we can't get the limit via some kind of system call but in bcbio Jeff brought this up and we swapped target_procs to match OSX default since the 50k number was an arbitrarily high number:

https://github.com/chapmanb/bcbio-nextgen/commit/0f590e12854df466053fcbfa590ab4ce9d7b9c45#diff-56930ee326340a3ab74bf8a0368e2d55

Hopefully this fix will get it working without issues for OSX testing. Thanks again.

[porterjamesj]

yeah that'll work. thanks!