Segfault: in dmax_k_SANDYBRIDGE ()

[looooo]

Issue:

Testing some function of dakota leads to a segfault which happens in liblapack. This is the backtrace:

#0  0x00007ffff0d653d7 in dmax_k_SANDYBRIDGE () from /home/lo/conda/envs/dakota_37_1/bin/../lib/liblapack.so.3
conda-forge/lapack-feedstock#1  0x00007fffeef21ebe in normal_ () from /home/lo/conda/envs/dakota_37_1/bin/../lib/./liblhs.so
conda-forge/lapack-feedstock#2  0x00007fffeef350ce in lhs_run_ () from /home/lo/conda/envs/dakota_37_1/bin/../lib/./liblhs.so
conda-forge/lapack-feedstock#3  0x00007ffff71d1eb8 in Pecos::LHSDriver::generate_samples(Teuchos::SerialDenseVector<int, double> const&, Teuchos::SerialDenseVector<int, double> const&, Teuchos::SerialDenseVector<int, int> const&, Teuchos::SerialDenseVector<int, int> const&, std::vector<std::set<int, std::less<int>, std::allocator<int> >, std::allocator<std::set<int, std::less<int>, std::allocator<int> > > > const&, std::vector<std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > const&, std::vector<std::set<double, std::less<double>, std::allocator<double> >, std::allocator<std::set<double, std::less<double>, std::allocator<double> > > > const&, Teuchos::SerialDenseVector<int, double> const&, Teuchos::SerialDenseVector<int, double> const&, Teuchos::SerialDenseVector<int, int> const&, Teuchos::SerialDenseVector<int, int> const&, std::vector<std::set<int, std::less<int>, std::allocator<int> >, std::allocator<std::set<int, std::less<int>, std::allocator<int> > > > const&, std::vector<std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > const&, std::vector<std::set<double, std::less<double>, std::allocator<double> >, std::allocator<std::set<double, std::less<double>, std::allocator<double> > > > const&, Pecos::AleatoryDistParams const&, Pecos::EpistemicDistParams const&, int, Teuchos::SerialDenseMatrix<int, double>&, Teuchos::SerialDenseMatrix<int, double>&) ()
   from /home/lo/conda/envs/dakota_37_1/bin/../lib/libpecos_src.so
conda-forge/lapack-feedstock#4  0x00007ffff79bfe3a in Dakota::NonDSampling::get_parameter_sets(Dakota::Model&, int, Teuchos::SerialDenseMatrix<int, double>&, bool) () from /home/lo/conda/envs/dakota_37_1/bin/../lib/libdakota_src.so
conda-forge/lapack-feedstock#5  0x00007ffff79d150d in Dakota::NonDLHSSampling::pre_run() () from /home/lo/conda/envs/dakota_37_1/bin/../lib/libdakota_src.so
conda-forge/lapack-feedstock#6  0x00007ffff78cc821 in Dakota::Iterator::run() () from /home/lo/conda/envs/dakota_37_1/bin/../lib/libdakota_src.so
conda-forge/lapack-feedstock#7  0x00007ffff7571954 in Dakota::IteratorScheduler::run_iterator(Dakota::Iterator&, std::_List_iterator<Dakota::ParallelLevel>) () from /home/lo/conda/envs/dakota_37_1/bin/../lib/libdakota_src.so
conda-forge/lapack-feedstock#8  0x00007ffff78c4716 in Dakota::Environment::execute() () from /home/lo/conda/envs/dakota_37_1/bin/../lib/libdakota_src.so
conda-forge/lapack-feedstock#9  0x00007ffff78c4862 in Dakota::ExecutableEnvironment::execute() () from /home/lo/conda/envs/dakota_37_1/bin/../lib/libdakota_src.so
conda-forge/lapack-feedstock#10 0x00005555555584ac in main ()

I don't think this will be solvable, but maybe someone has a hint whats going wrong here.

The same test worked with an older version of dakota and packages from conda-forge/label/cf201901.

[looooo]

btw.:

conda list

``` # Name Version Build Channel _libgcc_mutex 0.1 main appdirs 1.4.3 py_1 conda-forge arpack 3.6.3 h323e27b_1004 conda-forge attrs 19.1.0 py_0 conda-forge backcall 0.1.0 py_0 conda-forge blas 2.10 openblas conda-forge bleach 3.1.0 py_0 conda-forge boost 1.70.0 py37h9de70de_1 conda-forge boost-cpp 1.70.0 ha2d47e9_0 conda-forge bzip2 1.0.6 h14c3975_1002 conda-forge ca-certificates 2019.6.16 hecc5488_0 conda-forge calculix 2.15 h4c1d278_1002 conda-forge certifi 2019.6.16 py37_0 conda-forge clangdev 8.0.0 hc9558a2_2 conda-forge coin3d 4.0.0 h5b6a8d3_1006 conda-forge curl 7.64.1 hf8cf82a_0 conda-forge cycler 0.10.0 py_1 conda-forge dakota 6.10 py37hf2a3b5a_0 file:///home/lo/Diplomarbeit/dakota-conda-recipes/builds dakota-helpers 0.0 dev_0 dbus 1.13.6 he372182_0 conda-forge decorator 4.4.0 py_0 conda-forge defusedxml 0.5.0 py_1 conda-forge entrypoints 0.3 py37_1000 conda-forge expat 2.2.5 he1b5a44_1003 conda-forge ezdxf 0.9 py_0 conda-forge ezodf 0.3.2 py_2 conda-forge fontconfig 2.13.1 he4413a7_1000 conda-forge freecad 0.19.pre py37h0446ab6_26 freecad/label/dev freeimage 3.17.0 0 conda-forge freetype 2.10.0 he983fc9_0 conda-forge future 0.17.1 py37_1000 conda-forge gettext 0.19.8.1 hc5be6a0_1002 conda-forge glib 2.58.3 h6f030ca_1002 conda-forge gsl 2.5 h294904e_0 conda-forge gst-plugins-base 1.14.5 h0935bb2_0 conda-forge gstreamer 1.14.5 h36ae1b5_0 conda-forge hdf4 4.2.13 h9a582f1_1002 conda-forge hdf5 1.10.5 nompi_h3c11f04_1100 conda-forge icu 58.2 hf484d3e_1000 conda-forge ipykernel 5.1.1 py37h24bf2e0_0 conda-forge ipython 7.6.1 py37h5ca1d4c_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge jedi 0.14.1 py37_0 conda-forge jinja2 2.10.1 py_0 conda-forge joblib 0.13.2 py_0 conda-forge jpeg 9c h14c3975_1001 conda-forge json5 0.8.5 py_0 conda-forge jsoncpp 1.8.4 h6bb024c_1001 conda-forge jsonschema 3.0.1 py37_0 conda-forge jupyter_client 5.3.1 py_0 conda-forge jupyter_core 4.4.0 py_0 conda-forge jupyterlab 1.0.2 py37_0 conda-forge jupyterlab_server 1.0.0 py_1 conda-forge kiwisolver 1.1.0 py37hc9558a2_0 conda-forge krb5 1.16.3 h05b26f9_1001 conda-forge libblas 3.8.0 10_openblas conda-forge libcblas 3.8.0 10_openblas conda-forge libcurl 7.64.1 hda55be3_0 conda-forge libedit 3.1.20170329 hf8c457e_1001 conda-forge libffi 3.2.1 he1b5a44_1006 conda-forge libgcc-ng 9.1.0 hdf63c60_0 libgfortran-ng 7.3.0 hdf63c60_0 libglu 9.0.0 hf484d3e_1000 conda-forge libiconv 1.15 h516909a_1005 conda-forge liblapack 3.8.0 10_openblas conda-forge liblapacke 3.8.0 10_openblas conda-forge libnetcdf 4.6.2 h056eaf5_1002 conda-forge libopenblas 0.3.6 h6e990d7_4 conda-forge libpng 1.6.37 hed695b0_0 conda-forge libsodium 1.0.17 h516909a_0 conda-forge libspnav 0.2.3 h516909a_0 conda-forge libssh2 1.8.2 h22169c7_2 conda-forge libstdcxx-ng 9.1.0 hdf63c60_0 libtiff 4.0.10 h57b8799_1003 conda-forge libuuid 2.32.1 h14c3975_1000 conda-forge libxcb 1.13 h14c3975_1002 conda-forge libxml2 2.9.9 h13577e0_1 conda-forge libxslt 1.1.32 hae48121_1003 conda-forge llvmdev 8.0.0 hc9558a2_2 conda-forge lxml 4.3.4 py37h7ec2d77_0 conda-forge lz4-c 1.8.3 he1b5a44_1001 conda-forge markupsafe 1.1.1 py37h14c3975_0 conda-forge matplotlib-base 3.1.1 py37hfd891ef_0 conda-forge meshpy 2018.2.1 py37h637b7d7_1000 conda-forge mistune 0.8.4 py37h14c3975_1000 conda-forge mpi 1.0 openmpi conda-forge nbconvert 5.5.0 py_0 conda-forge nbformat 4.4.0 py_1 conda-forge ncurses 6.1 hf484d3e_1002 conda-forge netgen 6.2.1808 py37he2627c0_1006 conda-forge nomkl 3.0 0 notebook 5.7.8 py37_1 conda-forge numpy 1.16.4 py37h95a1406_0 conda-forge occt 7.3.0 h3be52bf_1004 conda-forge openblas 0.3.6 h6e990d7_4 conda-forge opencamlib 2018.08 py37h3e4de3e_0 freecad/label/dev openglider 0.1 dev_0 openmpi 4.0.1 hc99cbb1_1 conda-forge openssl 1.1.1c h516909a_0 conda-forge pandas 0.24.2 py37hb3f55d8_0 conda-forge pandoc 2.7.3 0 conda-forge pandocfilters 1.4.2 py_1 conda-forge parso 0.5.0 py_0 conda-forge patsy 0.5.1 py_0 conda-forge pcre 8.41 hf484d3e_1003 conda-forge pexpect 4.7.0 py37_0 conda-forge pickleshare 0.7.5 py37_1000 conda-forge pip 19.1.1 py37_0 conda-forge pivy 0.6.5a2 py37h4cb6e4d_2 conda-forge prometheus_client 0.7.1 py_0 conda-forge prompt_toolkit 2.0.9 py_0 conda-forge pthread-stubs 0.4 h14c3975_1001 conda-forge ptyprocess 0.6.0 py_1001 conda-forge py-xflr5 0.0.1 py37h4cb6e4d_1 openglider pyexcel-ezodf 0.3.4 pypi_0 pypi pygments 2.4.2 py_0 conda-forge pynastran 1.2.1 pypi_0 pypi pyparsing 2.4.0 py_0 conda-forge pyrsistent 0.15.3 py37h516909a_0 conda-forge pyside2 5.9.0a1 py37h4dc837a_0 conda-forge python 3.7.3 h33d41f4_1 conda-forge python-dateutil 2.8.0 py_0 conda-forge pytools 2019.1.1 py_0 conda-forge pytz 2019.1 py_0 conda-forge pyuff 0.0.1 py_1 file:///home/lo/Diplomarbeit/dakota-conda-recipes/builds pyyaml 5.1.1 py37h516909a_0 conda-forge pyzmq 18.0.2 py37hc4ba49a_1 conda-forge qt 5.9.7 h52cfd70_2 conda-forge readline 8.0 hf8c457e_0 conda-forge scikit-learn 0.21.2 py37hcdab131_1 conda-forge scipy 1.3.0 py37h921218d_0 conda-forge seaborn 0.9.0 py_1 conda-forge send2trash 1.5.0 py_0 conda-forge setuptools 41.0.1 py37_0 conda-forge six 1.12.0 py37_1000 conda-forge smesh 8.3.0.2 h5564c66_6 conda-forge soqt 1.6.0a h4cb6e4d_1002 conda-forge sqlite 3.29.0 hcee41ef_0 conda-forge statsmodels 0.10.0 py37hc1659b7_0 conda-forge svgwrite 1.3.1 py_0 conda-forge tbb 2019.7 hc9558a2_0 conda-forge terminado 0.8.2 py37_0 conda-forge testpath 0.4.2 py_1001 conda-forge tk 8.6.9 hed695b0_1002 conda-forge tornado 6.0.3 py37h516909a_0 conda-forge traitlets 4.3.2 py37_1000 conda-forge vtk 8.2.0 py37hfdee58b_203 conda-forge wcwidth 0.1.7 py_1 conda-forge webencodings 0.5.1 py_1 conda-forge wheel 0.33.4 py37_0 conda-forge xerces-c 3.2.2 hea5cb30_1002 conda-forge xorg-fixesproto 5.0 h14c3975_1002 conda-forge xorg-inputproto 2.3.2 h14c3975_1002 conda-forge xorg-kbproto 1.0.7 h14c3975_1002 conda-forge xorg-libice 1.0.10 h516909a_0 conda-forge xorg-libsm 1.2.3 h84519dc_1000 conda-forge xorg-libx11 1.6.7 h14c3975_1000 conda-forge xorg-libxau 1.0.9 h14c3975_0 conda-forge xorg-libxdmcp 1.1.3 h516909a_0 conda-forge xorg-libxext 1.3.4 h516909a_0 conda-forge xorg-libxfixes 5.0.3 h516909a_1004 conda-forge xorg-libxi 1.7.10 h516909a_0 conda-forge xorg-libxmu 1.1.3 h516909a_0 conda-forge xorg-libxt 1.1.5 h516909a_1003 conda-forge xorg-xextproto 7.3.0 h14c3975_1002 conda-forge xorg-xproto 7.0.31 h14c3975_1007 conda-forge xz 5.2.4 h14c3975_1001 conda-forge yaml 0.1.7 h14c3975_1001 conda-forge zeromq 4.3.2 he1b5a44_2 conda-forge zlib 1.2.11 h516909a_1005 conda-forge zstd 1.4.0 h3b9ef0a_0 conda-forge ```

conda info:

``` active environment : dakota_37_1 active env location : /home/lo/conda/envs/dakota_37_1 shell level : 2 user config file : /home/lo/.condarc populated config files : /home/lo/.condarc conda version : 4.7.11 conda-build version : 3.18.9 python version : 3.6.7.final.0 virtual packages : __cuda=9.1 base environment : /home/lo/conda (writable) channel URLs : https://conda.anaconda.org/freecad/label/dev/linux-64 https://conda.anaconda.org/freecad/label/dev/noarch https://conda.anaconda.org/openglider/linux-64 https://conda.anaconda.org/openglider/noarch file:///home/lo/Dropbox/conda-packages/linux-64 file:///home/lo/Dropbox/conda-packages/noarch file:///home/lo/Diplomarbeit/dakota-conda-recipes/builds/linux-64 file:///home/lo/Diplomarbeit/dakota-conda-recipes/builds/noarch https://conda.anaconda.org/conda-forge/linux-64 https://conda.anaconda.org/conda-forge/noarch https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /home/lo/conda/pkgs /home/lo/.conda/pkgs envs directories : /home/lo/conda/envs /home/lo/.conda/envs platform : linux-64 user-agent : conda/4.7.11 requests/2.22.0 CPython/3.6.7 Linux/5.0.0-27-generic ubuntu/18.04.3 glibc/2.27 UID:GID : 1000:1000 netrc file : None offline mode : False ```

[looooo]

I found a working package-combination which also includes a local build of dakota.

_libgcc_mutex             0.1                        main  
blas                      2.12                   openblas    conda-forge
boost-cpp                 1.68.0            h11c811c_1000    conda-forge
bzip2                     1.0.8                h516909a_1    conda-forge
ca-certificates           2019.6.16            hecc5488_0    conda-forge
certifi                   2019.6.16                py37_1    conda-forge
dakota                    6.10             py37h33ec4a3_1    file:///home/lo/Diplomarbeit/dakota-conda-recipes/builds
gsl                       2.4               h294904e_1006    conda-forge
icu                       58.2              hf484d3e_1000    conda-forge
libblas                   3.8.0               12_openblas    conda-forge
libcblas                  3.8.0               12_openblas    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_0  
liblapack                 3.8.0               12_openblas    conda-forge
liblapacke                3.8.0               12_openblas    conda-forge
libopenblas               0.3.7                h6e990d7_1    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0  
mpi                       1.0                     openmpi    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
nomkl                     3.0                           0  
numpy                     1.17.2           py37h95a1406_0    conda-forge
openblas                  0.3.7                h6e990d7_1    conda-forge
openmpi                   4.0.1                hc99cbb1_2    conda-forge
openssl                   1.1.1c               h516909a_0    conda-forge
pip                       19.2.3                   py37_0    conda-forge
python                    3.7.3                h33d41f4_1    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
setuptools                41.2.0                   py37_0    conda-forge
sqlite                    3.29.0               hcee41ef_1    conda-forge
tk                        8.6.9             hed695b0_1002    conda-forge
wheel                     0.33.6                   py37_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge

while this env doesn't work:

_libgcc_mutex             0.1                        main  
blas                      2.12                   openblas    conda-forge
boost-cpp                 1.70.0               h8e57a91_2    conda-forge
bzip2                     1.0.8                h516909a_1    conda-forge
ca-certificates           2019.6.16            hecc5488_0    conda-forge
certifi                   2019.6.16                py37_1    conda-forge
dakota                    6.10             py37hf2a3b5a_0    conda-forge
gsl                       2.5                  h294904e_1    conda-forge
icu                       64.2                 he1b5a44_1    conda-forge
libblas                   3.8.0               12_openblas    conda-forge
libcblas                  3.8.0               12_openblas    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_0  
liblapack                 3.8.0               12_openblas    conda-forge
liblapacke                3.8.0               12_openblas    conda-forge
libopenblas               0.3.7                h6e990d7_1    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0  
mpi                       1.0                     openmpi    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
nomkl                     3.0                           0  
numpy                     1.17.2           py37h95a1406_0    conda-forge
openblas                  0.3.7                h6e990d7_1    conda-forge
openmpi                   4.0.1                hc99cbb1_2    conda-forge
openssl                   1.1.1c               h516909a_0    conda-forge
pip                       19.2.3                   py37_0    conda-forge
python                    3.7.3                h33d41f4_1    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
setuptools                41.2.0                   py37_0    conda-forge
sqlite                    3.29.0               hcee41ef_1    conda-forge
tk                        8.6.9             hed695b0_1002    conda-forge
wheel                     0.33.6                   py37_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge

[looooo]

only building local does't solve the issue. There are only a few different package versions for these two environments:

-boost-cpp                 1.70.0               h8e57a91_2    conda-forge
+boost-cpp                 1.68.0            h11c811c_1000    conda-forge
-dakota                    6.10             py37hf2a3b5a_0    conda-forge
+dakota                    6.10             py37h33ec4a3_1    file:///home/lo/Diplomarbeit/dakota-conda-recipes/builds
-gsl                       2.5                  h294904e_1    conda-forge
+gsl                       2.4               h294904e_1006    conda-forge
-icu                       64.2                 he1b5a44_1    conda-forge
+icu                       58.2              hf484d3e_1000    conda-forge
-zlib                      1.2.11            h516909a_1006    conda-forge
+zlib                      1.2.11            h516909a_1006    conda-forge

[looooo]

@isuruf The build seems to use blas and lapack from netlib, but openblas is specified for the run-dependencies. Can this lead to such incompatibilities?

edit: Forcing netlib libraries [1] for the not working environment solved this issue. So I guess removing the openblas flag in the dakota-recipe will solve this issue.

[1] conda install liblapack=*=*netlib

[looooo]

@isuruf Is there a reason for using netlib instead of openblas by default for the build-env? Creating/updating environments always prefer openblas. So if this issue is not solvable, I think it would be best to switch these lines to openblas: https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/master/recipe/conda_build_config.yaml#L355L362

[isuruf]

See https://conda-forge.org/docs/maintainer/knowledge_base.html#how-it-works

[looooo]

Not sure I understand this, but in my eyes this issue is due to incompatibilities between netlib and openblas. Using openblas at build time fixed this issue. But I haven't tried to use netlib packages at run time. https://github.com/conda-forge/dakota-feedstock/pull/8/commits/587c6bf5f817506203cb163fae6deb27b58fcf04

[looooo]

btw.: I encountered another problem with netlib libraries. Somehow calling numpy (version 1.17) resulted in a crash. Using the same version as used for the build solved this issue. But as I am now building dakota with libraries from openblas, no such issue occures with numpy 1.17.

So somehow I have the impression libraries from openblas should be preffered over netlib libraries. I still don't know why netlib is default for the builds. Is there any reason?

[isuruf]

I still don't know why netlib is default for the builds. Is there any reason?

Please read the docs I linked above. Is there anything there that's not clear?

[isuruf]

Can you give steps to reproduce this?

[looooo]

Can you give steps to reproduce this?

I added a test-file to the dakota-feedstock. This should work with the current build because the build now uses openblas libraries, but I expect it to fail with the netlib libraries. I will report once the error is reproduced on the ci.

https://github.com/conda-forge/dakota-feedstock/pull/9

[looooo]

@isuruf I was able to reproduce the problem with azure: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=74324&view=logs

[isuruf]

The issue is that dmax symbol is defined in both openblas and dakota. When dakota is built with netlib and openblas is used, openblas' symbol takes precedence, but the two symbols do two different things. When dakota is built with openblas, somehow this issue goes away.

[looooo]

Thanks @isuruf for digging into this issue. If I have understood correctly this is a problem with dakota not using the right function. I will report this to the dakota-development team and stay with openblas for now.

[isuruf]

If I have understood correctly this is a problem with dakota not using the right function.

No, it's not. It's an issue with how we build packages using blas

[timsnyder]

@isuruf, thanks for the pointer to this ticket. Are you agreeing that we should just do the matrix build of blas_impl as I have done in conda-forge/dakota-feedstock#31?

@looooo did you ever bring this up with the upstream Dakota folks?