No way to limit the number of threads before importing / please honour OMP_NUM_THREADS

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Find a machine where ncores > ulimit -u
2. import numexpr

__init__.py checks for the number of cores on the machine and then calls 
set_num_threads(ncores), which seems to immediately spawn the number of threads 
requested. 

If the machine is a NUMA machine with a very large number of cores sharing the 
same memory, a very large number of threads may be created, which can be 
problematic, especially if the number of threads created is over ulimit -u.

In this case, python will exit with the message:

ERROR; return code from pthread_create() is 11                                  

(This is with numexpr 2.1 on an SGI Altix 1000 NUMA machine running RHEL 6.4.)

It would be nice if there were an environment variable to limit the number of 
threads created. The obvious example is OMP_NUM_THREADS, since it is set by 
most example job scripts on such machines. 

This can be done by the following patch to __init__.py, line 45.

import os

# Initialize the number of threads to be used
try:
    ncores = int(os.environ['OMP_NUM_THREADS'])
except KeyError:
    ncores = detect_number_of_cores()

Thanks.

Original issue reported on code.google.com by bla...@gmail.com on 17 Aug 2013 at 12:15

[GoogleCodeExporter]

Seconded -- with apparently no way to control the number of threads, this isn't 
really useable in a shared many-core environment, or when resources need to be 
managed

Original comment by dahi...@gmail.com on 22 Aug 2013 at 4:31

[GoogleCodeExporter]

This has been addressed in rev 02aa12848971.  Now, the number of threads is set 
to the number of cores or 8, whichever is less.  Also, the 
`NUMEXPR_NUM_THREADS` environment variable is honored.

This has been fully documented in Users Guide too:

https://code.google.com/p/numexpr/wiki/UsersGuide?ts=1377763727&updated=UsersGui
de#General_routines

Original comment by fal...@gmail.com on 29 Aug 2013 at 8:42

• Changed state: Fixed