search

ContinuumIO / anaconda-issues

Anaconda issue tracking
648 stars 224 forks source link

Critical MKL bug for `numpy.std` on Anaconda 5.2.0 on OSX #10089

Closed pberkes closed 4 months ago

pberkes commented 6 years ago

Actual Behavior

After importing a module compiled with fastmp, numpy.std returns wrong results.

Example:

Python 3.6.5 |Anaconda, Inc.| (default, Apr 26 2018, 08:42:37) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import numpy as np
   ...: import mklbug.do_nothing_fastmp
   ...: x = np.random.randn(1000000)
   ...: np.std(x)
   ...: 
   ...: 
Out[1]: 1644.8829343445466

The wrong behavior disappears when MKL_NUM_THREADS is set to 1.

The example attached demonstrate the bug using the operation the actually causes the bug: np.core.umath.multiply(x, x, out=x)

Expected Behavior

It should return approximately 1.0, and it does if one does not import the do_nothing_fastmp module.

Python 3.6.5 |Anaconda, Inc.| (default, Apr 26 2018, 08:42:37) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import numpy as np
   ...: x = np.random.randn(1000000)
   ...: np.std(x)
   ...: 
   ...: 
Out[1]: 1.000378135000183

Steps to Reproduce

0) Install Anaconda 5.2.0 and gcc 4.8.5 (using conda)

1) Unzip the attached package reproduce_bug.tar.gz

2) Run python setup.py develop This is going to compile two empty Cython modules: do_nothing_fastmp, which is compiled with -fopenmp, and do_nothing, with no extra compile options

3) Run python mklbug/run_me_fastmp.py to reproduce the bug

4) Run MKL_NUM_THREADS=1 python mklbug/run_me_fastmp.py to see the bug disappear

5) python mklbug/run_me.py does the same operations, but imports the module compiled without openmp, and it works as intended

This bug was first reported at https://github.com/velocyto-team/velocyto.py/issues/104 .

Anaconda or Miniconda version:

Anaconda 5.2.0, Python 3.6.5

Operating System:
conda info
Current conda install:

               platform : osx-64
          conda version : 4.3.30
       conda is private : False
      conda-env version : 4.3.30
    conda-build version : not installed
         python version : 3.5.2.final.0
       requests version : 2.14.2
       root environment : /Users/berkes/miniconda3  (writable)
    default environment : /Users/berkes/miniconda3/envs/bog
       envs directories : /Users/berkes/miniconda3/envs
                          /Users/berkes/.conda/envs
          package cache : /Users/berkes/miniconda3/pkgs
                          /Users/berkes/.conda/pkgs
           channel URLs : https://repo.continuum.io/pkgs/main/osx-64
                          https://repo.continuum.io/pkgs/main/noarch
                          https://repo.continuum.io/pkgs/free/osx-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/osx-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/osx-64
                          https://repo.continuum.io/pkgs/pro/noarch
            config file : /Users/berkes/.condarc
             netrc file : None
           offline mode : False
             user-agent : conda/4.3.30 requests/2.14.2 CPython/3.5.2 Darwin/17.4.0 OSX/10.13.3    
                UID:GID : 735983909:1116983699
conda list --show-channel-urls
# packages in environment at /Users/berkes/miniconda3/envs/bog:
#
alabaster                 0.7.10           py36h174008c_0    defaults
anaconda                  5.2.0                    py36_3    defaults
anaconda-client           1.6.14                   py36_0    defaults
anaconda-project          0.8.2            py36h9ee5d53_0    defaults
appnope                   0.1.0            py36hf537a9a_0    defaults
appscript                 1.0.1            py36h9e71e49_1    defaults
asn1crypto                0.24.0                   py36_0    defaults
astroid                   1.6.3                    py36_0    defaults
astropy                   3.0.2            py36h917ab60_1    defaults
attrs                     18.1.0                   py36_0    defaults
babel                     2.5.3                    py36_0    defaults
backcall                  0.1.0                    py36_0    defaults
backports                 1.0              py36ha3c1827_1    defaults
backports.shutil_get_terminal_size 1.0.0            py36hd7a2ee4_2    defaults
beautifulsoup4            4.6.0            py36h72d3c9f_1    defaults
bitarray                  0.8.1            py36h1de35cc_1    defaults
bkcharts                  0.2              py36h073222e_0    defaults
blas                      1.0                         mkl    defaults
blaze                     0.11.3           py36h02e7a37_0    defaults
bleach                    2.1.3                    py36_0    defaults
blosc                     1.14.3               hd9629dc_0    defaults
bokeh                     0.12.16                  py36_0    defaults
boto                      2.48.0           py36hdbc59ac_1    defaults
bottleneck                1.2.1            py36hbd380ad_0    defaults
bzip2                     1.0.6                h1de35cc_5    defaults
ca-certificates           2018.03.07                    0    defaults
certifi                   2018.4.16                py36_0    defaults
cffi                      1.11.5           py36h342bebf_0    defaults
chardet                   3.0.4            py36h96c241c_1    defaults
click                     6.7              py36hec950be_0    defaults
cloog                     0.18.0                        0    defaults
cloudpickle               0.5.3                    py36_0    defaults
clyent                    1.2.2            py36hae3ad88_0    defaults
colorama                  0.3.9            py36hd29a30c_0    defaults
contextlib2               0.5.5            py36hd66e5e7_0    defaults
cryptography              2.2.2            py36h1de35cc_0    defaults
curl                      7.60.0               ha441bb4_0    defaults
cycler                    0.10.0           py36hfc81398_0    defaults
cython                    0.28.2           py36h1de35cc_0    defaults
cytoolz                   0.9.0.1          py36h1de35cc_0    defaults
dask                      0.17.5                   py36_0    defaults
dask-core                 0.17.5                   py36_0    defaults
datashape                 0.5.4            py36hfb22df8_0    defaults
dbus                      1.13.2               h760590f_1    defaults
decorator                 4.3.0                    py36_0    defaults
distributed               1.21.8                   py36_0    defaults
docutils                  0.14             py36hbfde631_0    defaults
entrypoints               0.2.3            py36hd81d71f_2    defaults
et_xmlfile                1.0.1            py36h1315bdc_0    defaults
expat                     2.2.5                hb8e80ba_0    defaults
fastcache                 1.0.2            py36h1de35cc_2    defaults
filelock                  3.0.4                    py36_0    defaults
flask                     1.0.2                    py36_1    defaults
flask-cors                3.0.4                    py36_0    defaults
freetype                  2.8                  h12048fb_1    defaults
gcc                       4.8.5                         8    defaults
get_terminal_size         1.0.0                h7520d66_0    defaults
gettext                   0.19.8.1             h15daf44_3    defaults
gevent                    1.3.0            py36h1de35cc_0    defaults
glib                      2.56.1               h35bc53a_0    defaults
glob2                     0.6              py36h94c9186_0    defaults
gmp                       6.1.2                hb37e062_1    defaults
gmpy2                     2.0.8            py36hf9c35bd_2    defaults
greenlet                  0.4.13           py36h1de35cc_0    defaults
h5py                      2.7.1            py36ha8ecd60_2    defaults
hdf5                      1.10.2               hfa1e0ec_1    defaults
heapdict                  1.0.0                    py36_2    defaults
html5lib                  1.0.1            py36h2f9c1c0_0    defaults
icu                       58.2                 h4b95b61_1    defaults
idna                      2.6              py36h8628d0a_1    defaults
imageio                   2.3.0                    py36_0    defaults
imagesize                 1.0.0                    py36_0    defaults
intel-openmp              2018.0.0                      8    defaults
ipykernel                 4.8.2                    py36_0    defaults
ipython                   6.4.0                    py36_0    defaults
ipython_genutils          0.2.0            py36h241746c_0    defaults
ipywidgets                7.2.1                    py36_0    defaults
isl                       0.12.2                        1    defaults
isort                     4.3.4                    py36_0    defaults
itsdangerous              0.24             py36h49fbb8d_1    defaults
jbig                      2.1                  h4d881f8_0    defaults
jdcal                     1.4                      py36_0    defaults
jedi                      0.12.0                   py36_1    defaults
jinja2                    2.10             py36hd36f9c5_0    defaults
jpeg                      9b                   he5867d9_2    defaults
jsonschema                2.6.0            py36hb385e00_0    defaults
jupyter                   1.0.0                    py36_4    defaults
jupyter_client            5.2.3                    py36_0    defaults
jupyter_console           5.2.0            py36hccf5b1c_1    defaults
jupyter_core              4.4.0            py36h79cf704_0    defaults
jupyterlab                0.32.1                   py36_0    defaults
jupyterlab_launcher       0.10.5                   py36_0    defaults
kiwisolver                1.0.1            py36h792292d_0    defaults
lazy-object-proxy         1.3.1            py36h2fbbe47_0    defaults
libcurl                   7.60.0               hf30b1f0_0    defaults
libcxx                    4.0.1                h579ed51_0    defaults
libcxxabi                 4.0.1                hebd6815_0    defaults
libedit                   3.1.20170329         hb402a30_2    defaults
libffi                    3.2.1                h475c297_4    defaults
libgfortran               3.0.1                h93005f0_2    defaults
libiconv                  1.15                 hdd342a3_7    defaults
libpng                    1.6.34               he12f830_0    defaults
libsodium                 1.0.16               h3efe00b_0    defaults
libssh2                   1.8.0                h322a93b_4    defaults
libtiff                   4.0.9                hcb84e12_1    defaults
libxml2                   2.9.8                hab757c2_1    defaults
libxslt                   1.1.32               hb819dd2_0    defaults
llvmlite                  0.23.1           py36hc454e04_0    defaults
locket                    0.2.0            py36hca03003_1    defaults
loompy                    2.0.12                    <pip>
lxml                      4.2.1            py36h7166777_0    defaults
lzo                       2.10                 h362108e_2    defaults
markupsafe                1.0              py36h3a1e703_1    defaults
matplotlib                2.2.2            py36ha7267d0_0    defaults
mccabe                    0.6.1            py36hdaeb55d_0    defaults
mistune                   0.8.3            py36h1de35cc_1    defaults
mkl                       2018.0.2                      1    defaults
mkl-service               1.1.2            py36h7ea6df4_4    defaults
mkl_fft                   1.0.1            py36h917ab60_0    defaults
mkl_random                1.0.1            py36h78cc56f_0    defaults
more-itertools            4.1.0                    py36_0    defaults
mpc                       1.0.3                h7a72875_5    defaults
mpfr                      3.1.5                h711e7fd_2    defaults
mpmath                    1.0.0            py36hf1b8295_2    defaults
msgpack-python            0.5.6            py36h04f5b5a_0    defaults
multipledispatch          0.5.0                    py36_0    defaults
nbconvert                 5.3.1            py36h810822e_0    defaults
nbformat                  4.4.0            py36h827af21_0    defaults
ncurses                   6.1                  h0a44026_0    defaults
networkx                  2.1                      py36_0    defaults
nltk                      3.3.0                    py36_0    defaults
nose                      1.3.7            py36h73fae2b_2    defaults
notebook                  5.5.0                    py36_0    defaults
numba                     0.38.0           py36h1702cab_0    defaults
numexpr                   2.6.5            py36h057f876_0    defaults
numpy                     1.14.3           py36h9bb19eb_1    defaults
numpy-base                1.14.3           py36h479e554_1    defaults
numpydoc                  0.8.0                    py36_0    defaults
odo                       0.5.1            py36hc1af34a_0    defaults
olefile                   0.45.1                   py36_0    defaults
openpyxl                  2.5.3                    py36_0    defaults
openssl                   1.0.2o               h26aff7b_0    defaults
packaging                 17.1                     py36_0    defaults
pandas                    0.23.0           py36h1702cab_0    defaults
pandoc                    1.19.2.1             ha5e8f32_1    defaults
pandocfilters             1.4.2            py36h3b0b094_1    defaults
parso                     0.2.0                    py36_0    defaults
partd                     0.3.8            py36hf5c4cb8_0    defaults
path.py                   11.0.1                   py36_0    defaults
pathlib2                  2.3.2                    py36_0    defaults
patsy                     0.5.0                    py36_0    defaults
pcre                      8.42                 h378b8a2_0    defaults
pep8                      1.7.1                    py36_0    defaults
pexpect                   4.5.0                    py36_0    defaults
pickleshare               0.7.4            py36hf512f8e_0    defaults
pillow                    5.1.0            py36hfcce615_0    defaults
pip                       10.0.1                   py36_0    defaults
pkginfo                   1.4.2                    py36_1    defaults
pluggy                    0.6.0            py36hb1d0581_0    defaults
ply                       3.11                     py36_0    defaults
prompt_toolkit            1.0.15           py36haeda067_0    defaults
psutil                    5.4.5            py36h1de35cc_0    defaults
ptyprocess                0.5.2            py36he6521c3_0    defaults
py                        1.5.3                    py36_0    defaults
pycodestyle               2.4.0                    py36_0    defaults
pycosat                   0.6.3            py36hee92d8f_0    defaults
pycparser                 2.18             py36h724b2fc_1    defaults
pycrypto                  2.6.1            py36h1de35cc_8    defaults
pycurl                    7.43.0.1         py36hdbc3d79_0    defaults
pyflakes                  1.6.0            py36hea45e83_0    defaults
pygments                  2.2.0            py36h240cd3f_0    defaults
pylint                    1.8.4                    py36_0    defaults
pyodbc                    4.0.23           py36h0a44026_0    defaults
pyopenssl                 18.0.0                   py36_0    defaults
pyparsing                 2.2.0            py36hb281f35_0    defaults
pyqt                      5.9.2            py36h11d3b92_0    defaults
pysam                     0.15.1                    <pip>
pysocks                   1.6.8                    py36_0    defaults
pytables                  3.4.3            py36h5ca999c_2    defaults
pytest                    3.5.1                    py36_0    defaults
pytest-arraydiff          0.2                      py36_0    defaults
pytest-astropy            0.3.0                    py36_0    defaults
pytest-doctestplus        0.1.3                    py36_0    defaults
pytest-openfiles          0.3.0                    py36_0    defaults
pytest-remotedata         0.2.1                    py36_0    defaults
python                    3.6.5                hc167b69_1    defaults
python-dateutil           2.7.3                    py36_0    defaults
python.app                2                        py36_8    defaults
pytz                      2018.4                   py36_0    defaults
pywavelets                0.5.2            py36h2710a04_0    defaults
pyyaml                    3.12             py36h2ba1e63_1    defaults
pyzmq                     17.0.0           py36h1de35cc_1    defaults
qt                        5.9.5                h02808f3_0    defaults
qtawesome                 0.4.4            py36h468c6fb_0    defaults
qtconsole                 4.3.1            py36hd96c0ff_0    defaults
qtpy                      1.4.1                    py36_0    defaults
readline                  7.0                  hc1231fa_4    defaults
requests                  2.18.4           py36h4516966_1    defaults
rope                      0.10.7           py36h68959ac_0    defaults
ruamel_yaml               0.15.35          py36h1de35cc_1    defaults
scikit-image              0.13.1           py36h1de35cc_1    defaults
scikit-learn              0.19.1           py36hffbff8c_0    defaults
scipy                     1.1.0            py36hcaad992_0    defaults
seaborn                   0.8.1            py36h595ecd9_0    defaults
send2trash                1.5.0                    py36_0    defaults
setuptools                39.1.0                   py36_0    defaults
simplegeneric             0.8.1                    py36_2    defaults
singledispatch            3.4.0.3          py36hf20db9d_0    defaults
sip                       4.19.8           py36h0a44026_0    defaults
six                       1.11.0           py36h0e22d5e_1    defaults
snappy                    1.1.7                he62c110_3    defaults
snowballstemmer           1.2.1            py36h6c7b616_0    defaults
sortedcollections         0.6.1                    py36_0    defaults
sortedcontainers          1.5.10                   py36_0    defaults
sphinx                    1.7.4                    py36_0    defaults
sphinxcontrib             1.0              py36h9364dc8_1    defaults
sphinxcontrib-websupport  1.0.1            py36h92f4a7a_1    defaults
spyder                    3.2.8                    py36_0    defaults
sqlalchemy                1.2.7            py36hb402a30_0    defaults
sqlite                    3.23.1               hf1716c9_0    defaults
statsmodels               0.9.0            py36h917ab60_0    defaults
sympy                     1.1.1            py36h7f3cf04_0    defaults
tblib                     1.3.2            py36hda67792_0    defaults
terminado                 0.8.1                    py36_1    defaults
testpath                  0.3.1            py36h625a49b_0    defaults
tk                        8.6.7                h35a86e2_3    defaults
toolz                     0.9.0                    py36_0    defaults
tornado                   5.0.2                    py36_0    defaults
traitlets                 4.3.2            py36h65bd3ce_0    defaults
typing                    3.6.4                    py36_0    defaults
unicodecsv                0.14.1           py36he531d66_0    defaults
unixodbc                  2.3.6                h3efe00b_0    defaults
urllib3                   1.22             py36h68b9469_0    defaults
wcwidth                   0.1.7            py36h8c6ec74_0    defaults
webencodings              0.5.1            py36h3b9701d_1    defaults
werkzeug                  0.14.1                   py36_0    defaults
wheel                     0.31.1                   py36_0    defaults
widgetsnbextension        3.2.1                    py36_0    defaults
wrapt                     1.10.11          py36hc29e774_0    defaults
xlrd                      1.1.0            py36h336f4a2_1    defaults
xlsxwriter                1.0.4                    py36_0    defaults
xlwings                   0.11.8                   py36_0    defaults
xlwt                      1.2.0            py36h5ad1178_0    defaults
xz                        5.2.4                h1de35cc_4    defaults
yaml                      0.1.7                hc338f04_2    defaults
zeromq                    4.2.5                h378b8a2_0    defaults
zict                      0.1.3            py36h71da714_0    defaults
zlib                      1.2.11               hf3cbc9b_2    defaults
pberkes commented 6 years ago

Note that the bug disappear on Anaconda 5.0.1:

Python 3.6.3 |Anaconda, Inc.| (default, Oct  6 2017, 12:04:38) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import numpy as np
   ...: import mklbug.do_nothing_fastmp
   ...: x = np.random.randn(1000000)
   ...: np.std(x)
   ...: 
Out[1]: 1.001603803553599
msarahan commented 6 years ago

Try updating numpy. There is a race condition with OpenMP implementations. MKL assumes intel's implementation by default unless you tell it otherwise. Later numpy builds have some runtime detection that Intel contributed, and that helps iron this out. It seems like your version numpy should be new enough to have this, but maybe there's something else going wrong

More discussion at

pberkes commented 6 years ago

Hi @msarahan, thanks for the quick reply. I can confirm that updating numpy resolves the issue.

OTOH, users are installing the latest version of Anaconda to run popular scientific packages and are getting wrong results (as in https://github.com/velocyto-team/velocyto.py/issues/104).

So I guess the bug is that the set of versions in Anaconda should be updated so that this does not occur? Would you like me to close this ticket and open a new one?