Jobs failing due to OpenBLAS

calchoo commented 5 years ago

MAVIS version: 2.2.3

Python version: 3.6.0

OS: Centos 6.7

Jobs are failing with thread count errors due to OpenBLAS (a part of numpy):

OpenBLAS blas_thread_init: pthread_create failed for thread 8 of 48: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 1549338 max
OpenBLAS blas_thread_init: pthread_create failed for thread 9 of 48: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 1549338 max
OpenBLAS blas_thread_init: pthread_create failed for thread 10 of 48: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 1549338 max
OpenBLAS blas_thread_init: pthread_create failed for thread 11 of 48: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 1549338 max
Traceback (most recent call last):
  File "/gsc/pipelines/mavis/v2.2.3/venv/bin/mavis", line 11, in <module>
    load_entry_point('mavis==2.2.3', 'console_scripts', 'mavis')()
  File "/gsc/pipelines/mavis/v2.2.3/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/gsc/pipelines/mavis/v2.2.3/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2843, in load_entry_point
    return ep.load()
  File "/gsc/pipelines/mavis/v2.2.3/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2434, in load
    return self.resolve()
  File "/gsc/pipelines/mavis/v2.2.3/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2440, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/gsc/pipelines/mavis/v2.2.3/venv/lib/python3.6/site-packages/mavis-2.2.3-py3.6.egg/mavis/main.py", line 17, in <module>
    from . import config as _config
  File "/gsc/pipelines/mavis/v2.2.3/venv/lib/python3.6/site-packages/mavis-2.2.3-py3.6.egg/mavis/config.py", line 17, in <module>
    from .bam.stats import compute_genome_bam_stats, compute_transcriptome_bam_stats
  File "/gsc/pipelines/mavis/v2.2.3/venv/lib/python3.6/site-packages/mavis-2.2.3-py3.6.egg/mavis/bam/stats.py", line 6, in <module>
    import numpy as np
  File "/gsc/pipelines/mavis/v2.2.3/venv/lib/python3.6/site-packages/numpy-1.16.2-py3.6-linux-x86_64.egg/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/gsc/pipelines/mavis/v2.2.3/venv/lib/python3.6/site-packages/numpy-1.16.2-py3.6-linux-x86_64.egg/numpy/core/__init__.py", line 40, in <module>
    from . import multiarray
  File "/gsc/pipelines/mavis/v2.2.3/venv/lib/python3.6/site-packages/numpy-1.16.2-py3.6-linux-x86_64.egg/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/gsc/pipelines/mavis/v2.2.3/venv/lib/python3.6/site-packages/numpy-1.16.2-py3.6-linux-x86_64.egg/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
KeyboardInterrupt

This seems to happen when too many jobs are running at the same time and when mavis is using numpy v1.16.0+.

calchoo commented 5 years ago

It looks like the newer numpy versions are using an openBLAS version that spawns a thread count equal to the number of processors on the machine by default.

Possible fixes include:

explicitly setting the number of openBLAS threads using: export OMP_NUM_THREADS=1
restricting the numpy version to be <1.16.0

creisle commented 5 years ago

Is there a way to set this from where we import numpy? or does it have to be the environment variable? it will be hard to incorporate the env variable and ensure it is used

creisle commented 5 years ago

Looks like maybe? https://stackoverflow.com/questions/30791550/limit-number-of-threads-in-numpy

Try setting the env variable just before we import numpy

calchoo commented 5 years ago

yeah, it looks like we could set it in the script itself

import os
os.environ["OMP_NUM_THREADS"] = "4" # export OMP_NUM_THREADS=4
os.environ["OPENBLAS_NUM_THREADS"] = "4" # export OPENBLAS_NUM_THREADS=4 
os.environ["MKL_NUM_THREADS"] = "6" # export MKL_NUM_THREADS=6
os.environ["VECLIB_MAXIMUM_THREADS"] = "4" # export VECLIB_MAXIMUM_THREADS=4
os.environ["NUMEXPR_NUM_THREADS"] = "6" # export NUMEXPR_NUM_THREADS=6

calchoo commented 5 years ago

Fixed.

bcgsc / mavis