Segmentation fault with numpy.fft.rfft and MKL and threads

[LukasDrude]

When using numpy.fft.rfft with mkl_fft in a ThreadPoolExecutor you can run in a segmentation fault.

Why is this a cross-post? Since this issue is related to Anaconda, Numpy and mkl_fft, this issue is posted on all three locations. Currently, it is not quite clear which party should address the issue.

• mkl_fft: No response yet.
• numpy: Sees issue at mkl_fft but uses a seemingly buggy library (namely mkl_fft).
• anaconda: Here.

We get the same error when testing this on different infrastructure/ OS:

• Scientific Linux 7.2 (Nitrogen): anaconda version 5.2.0 build py36_3
• Ubuntu 16.04.4 LTS: anaconda version custom build py36hbbc8b67_0
• OSX: anaconda version 5.2.0 build py36_3

import numpy as np
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

def fun(_):
    print(_)
    frames = np.random.randint(1000, 2000)
    signal = np.ones((2, frames), dtype=np.float64)

    _ = np.fft.rfft(signal)

    return 1,

if __name__ == '__main__':
    # print('ProcessPoolExecutor')
    # with ProcessPoolExecutor(4) as ex:
    #     list(ex.map(fun, range(50)))

    print('ThreadPoolExecutor')
    with ThreadPoolExecutor(4) as ex:
        list(ex.map(fun, range(500)))

Repeating the example code many times yields some kind of traceback, if it did not segfault.

Sometimes occuring traceback

``` Traceback (most recent call last): File "test_script.py", line 35, in list(ex.map(fun, range(50))) File "/net/home/pyadmin/conda/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator yield fs.pop().result() File "/net/home/pyadmin/conda/lib/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/net/home/pyadmin/conda/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/net/home/pyadmin/conda/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "test_script.py", line 11, in fun _ = np.fft.rfft(signal) File "/net/home/pyadmin/conda/lib/python3.6/site-packages/mkl_fft/_numpy_fft.py", line 331, in rfft output = mkl_fft.rfft_numpy(a, n=n, axis=axis) File "mkl_fft/_pydfti.pyx", line 569, in mkl_fft._pydfti.rfft_numpy File "mkl_fft/_pydfti.pyx", line 487, in mkl_fft._pydfti._rc_fft1d_impl ValueError: Internal error occurred ```

Not working environment:

Conda installed numpy version: Version: 1.14.3

Numpy config

``` python -c "import numpy; print(numpy.show_config())" mkl_info: libraries = ['mkl_rt', 'pthread'] library_dirs = ['/Users/lukas/anaconda3/lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['/Users/lukas/anaconda3/include'] blas_mkl_info: libraries = ['mkl_rt', 'pthread'] library_dirs = ['/Users/lukas/anaconda3/lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['/Users/lukas/anaconda3/include'] blas_opt_info: libraries = ['mkl_rt', 'pthread'] library_dirs = ['/Users/lukas/anaconda3/lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['/Users/lukas/anaconda3/include'] lapack_mkl_info: libraries = ['mkl_rt', 'pthread'] library_dirs = ['/Users/lukas/anaconda3/lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['/Users/lukas/anaconda3/include'] lapack_opt_info: libraries = ['mkl_rt', 'pthread'] library_dirs = ['/Users/lukas/anaconda3/lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['/Users/lukas/anaconda3/include'] None ```

Working Environment:

Pip installed numpy version: 1.14.5

Numpy config

``` python -c "import numpy; print(numpy.show_config())" blas_mkl_info: NOT AVAILABLE blis_info: NOT AVAILABLE openblas_info: NOT AVAILABLE atlas_3_10_blas_threads_info: NOT AVAILABLE atlas_3_10_blas_info: NOT AVAILABLE atlas_blas_threads_info: NOT AVAILABLE atlas_blas_info: NOT AVAILABLE blas_opt_info: extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers'] extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)] lapack_mkl_info: NOT AVAILABLE openblas_lapack_info: NOT AVAILABLE openblas_clapack_info: NOT AVAILABLE atlas_3_10_threads_info: NOT AVAILABLE atlas_3_10_info: NOT AVAILABLE atlas_threads_info: NOT AVAILABLE atlas_info: NOT AVAILABLE lapack_opt_info: extra_compile_args = ['-msse3'] extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)] None ```

[boeddeker]

I found that

import numpy as np
np.fft.restore_all()

is a workaround until anaconda disables mkl_fft.

[boeddeker]

Why is mkl_fft not disable by default in anaconda? In https://github.com/IntelPython/mkl_fft/issues/24 is denoted that the bug is still there.