undefined symbol: dorgqr_

[echristophe]

I'm running into issues when importing fdasrsf: undefined symbol: dorgqr_

• same issue whether it's install via pip or via python setup.py

Full error:

Python 3.7.5rc1 (default, Oct  8 2019, 16:47:45) 
[GCC 9.2.1 20190909] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fdasrsf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/echristophe/.local/lib/python3.7/site-packages/fdasrsf-2.0.0-py3.7-linux-x86_64.egg/fdasrsf/__init__.py", line 22, in <module>
    from .time_warping import fdawarp, align_fPCA, align_fPLS, pairwise_align_bayes
  File "/home/echristophe/.local/lib/python3.7/site-packages/fdasrsf-2.0.0-py3.7-linux-x86_64.egg/fdasrsf/time_warping.py", line 9, in <module>
    import fdasrsf.utility_functions as uf
  File "/home/echristophe/.local/lib/python3.7/site-packages/fdasrsf-2.0.0-py3.7-linux-x86_64.egg/fdasrsf/utility_functions.py", line 20, in <module>
    import optimum_reparam_Ng as orNg
ImportError: /home/echristophe/.local/lib/python3.7/site-packages/fdasrsf-2.0.0-py3.7-linux-x86_64.egg/optimum_reparam_Ng.cpython-37m-x86_64-linux-gnu.so: undefined symbol: dorgqr_

Here is the part of the compilation process related to optimum_reparam_Ng:

skipping 'src/optimum_reparam_Ng.cpp' Cython extension (up-to-date)
building 'optimum_reparam_Ng' extension
creating build/temp.linux-x86_64-3.7/src/gropt
creating build/temp.linux-x86_64-3.7/src/gropt/src
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/lib/python3/dist-packages/numpy/core/include -Isrc/gropt/incl/ -I/usr/include/python3.7m -c src/optimum_reparam_Ng.cpp -o build/temp.linux-x86_64-3.7/src/optimum_reparam_Ng.o -std=c++11

And linking:

x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/src/optimum_reparam_Ng.o build/temp.linux-x86_64-3.7/src/gropt/src/MyMatrix.o build/temp.linux-x86_64-3.7/src/gropt/src/SphereVector.o build/temp.linux-x86_64-3.7/src/gropt/src/SphereVariable.o build/temp.linux-x86_64-3.7/src/gropt/src/Euclidean.o build/temp.linux-x86_64-3.7/src/gropt/src/Oblique.o build/temp.linux-x86_64-3.7/src/gropt/src/EucVariable.o build/temp.linux-x86_64-3.7/src/gropt/src/LowRankVector.o build/temp.linux-x86_64-3.7/src/gropt/src/ElasticCurvesRO.o build/temp.linux-x86_64-3.7/src/gropt/src/ObliqueVector.o build/temp.linux-x86_64-3.7/src/gropt/src/StieVector.o build/temp.linux-x86_64-3.7/src/gropt/src/nist_spblas.o build/temp.linux-x86_64-3.7/src/gropt/src/RTRSD.o build/temp.linux-x86_64-3.7/src/gropt/src/randgen.o build/temp.linux-x86_64-3.7/src/gropt/src/SolversLS.o build/temp.linux-x86_64-3.7/src/gropt/src/QuasiNewton.o build/temp.linux-x86_64-3.7/src/gropt/src/DriverElasticCurvesRO.o build/temp.linux-x86_64-3.7/src/gropt/src/Manifold.o build/temp.linux-x86_64-3.7/src/gropt/src/OrthGroup.o build/temp.linux-x86_64-3.7/src/gropt/src/OrthGroupVariable.o build/temp.linux-x86_64-3.7/src/gropt/src/SphereConvexHull.o build/temp.linux-x86_64-3.7/src/gropt/src/LRBFGS.o build/temp.linux-x86_64-3.7/src/gropt/src/ProductElement.o build/temp.linux-x86_64-3.7/src/gropt/src/Element.o build/temp.linux-x86_64-3.7/src/gropt/src/RTRSR1.o build/temp.linux-x86_64-3.7/src/gropt/src/EucFrechetMean.o build/temp.linux-x86_64-3.7/src/gropt/src/RWRBFGS.o build/temp.linux-x86_64-3.7/src/gropt/src/Timer.o build/temp.linux-x86_64-3.7/src/gropt/src/OrthGroupVector.o build/temp.linux-x86_64-3.7/src/gropt/src/SharedSpace.o build/temp.linux-x86_64-3.7/src/gropt/src/SolversTR.o build/temp.linux-x86_64-3.7/src/gropt/src/RSD.o build/temp.linux-x86_64-3.7/src/gropt/src/Solvers.o build/temp.linux-x86_64-3.7/src/gropt/src/LRTRSR1.o build/temp.linux-x86_64-3.7/src/gropt/src/RNewton.o build/temp.linux-x86_64-3.7/src/gropt/src/GrassVariable.o build/temp.linux-x86_64-3.7/src/gropt/src/EucVector.o build/temp.linux-x86_64-3.7/src/gropt/src/Grassmann.o build/temp.linux-x86_64-3.7/src/gropt/src/ElasticCurvesReparam.o build/temp.linux-x86_64-3.7/src/gropt/src/Sphere.o build/temp.linux-x86_64-3.7/src/gropt/src/ProductManifold.o build/temp.linux-x86_64-3.7/src/gropt/src/SmartSpace.o build/temp.linux-x86_64-3.7/src/gropt/src/Spline.o build/temp.linux-x86_64-3.7/src/gropt/src/LowRank.o build/temp.linux-x86_64-3.7/src/gropt/src/L2SphereVariable.o build/temp.linux-x86_64-3.7/src/gropt/src/RBFGS.o build/temp.linux-x86_64-3.7/src/gropt/src/LowRankVariable.o build/temp.linux-x86_64-3.7/src/gropt/src/RBroydenFamily.o build/temp.linux-x86_64-3.7/src/gropt/src/Problem.o build/temp.linux-x86_64-3.7/src/gropt/src/ObliqueVariable.o build/temp.linux-x86_64-3.7/src/gropt/src/L2Sphere.o build/temp.linux-x86_64-3.7/src/gropt/src/L2SphereVector.o build/temp.linux-x86_64-3.7/src/gropt/src/StieVariable.o build/temp.linux-x86_64-3.7/src/gropt/src/GrassVector.o build/temp.linux-x86_64-3.7/src/gropt/src/RCG.o build/temp.linux-x86_64-3.7/src/gropt/src/Stiefel.o build/temp.linux-x86_64-3.7/src/gropt/src/RTRNewton.o build/temp.linux-x86_64-3.7/src/gropt/src/LinearOPE.o build/temp.linux-x86_64-3.7/src/gropt/src/MinPNormConHull.o build/temp.linux-x86_64-3.7/src/gropt/src/ForDebug.o -o build/lib.linux-x86_64-3.7/optimum_reparam_Ng.cpython-37m-x86_64-linux-gnu.so -std=c++11

[jdtuck]

Can you please provide more information, i.e., Linux distro, kernel, what BLAS you have installed. Also can you let me know what blas that numpy is linking to. I would like to recreate this, but cannot on my two current Linux builds

[echristophe]

Debian testing, kernel 5.2.17

libblass and libblas64-3 are 3.8.0-7 numpy is the distribution package (1:1.16.5-1) so I expect it to be linking to the libblass above.

[jdtuck]

Can you please provide the output of numpy.config.show()

[echristophe]

>>> numpy.__config__.show()
blas_mkl_info:
  NOT AVAILABLE
blis_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
atlas_3_10_blas_threads_info:
  NOT AVAILABLE
atlas_3_10_blas_info:
  NOT AVAILABLE
atlas_blas_threads_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
accelerate_info:
  NOT AVAILABLE
blas_info:
    libraries = ['blas', 'blas']
    library_dirs = ['/usr/lib/x86_64-linux-gnu']
    include_dirs = ['/usr/local/include', '/usr/include']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
blas_opt_info:
    define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
    libraries = ['blas', 'blas']
    library_dirs = ['/usr/lib/x86_64-linux-gnu']
    include_dirs = ['/usr/local/include', '/usr/include']
    language = c
lapack_mkl_info:
  NOT AVAILABLE
openblas_lapack_info:
  NOT AVAILABLE
openblas_clapack_info:
  NOT AVAILABLE
atlas_3_10_threads_info:
  NOT AVAILABLE
atlas_3_10_info:
  NOT AVAILABLE
atlas_threads_info:
  NOT AVAILABLE
atlas_info:
  NOT AVAILABLE
lapack_info:
    libraries = ['lapack', 'lapack']
    library_dirs = ['/usr/lib/x86_64-linux-gnu']
    language = f77
lapack_opt_info:
    define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
    libraries = ['lapack', 'lapack', 'blas', 'blas']
    library_dirs = ['/usr/lib/x86_64-linux-gnu']
    language = c
    include_dirs = ['/usr/local/include', '/usr/include']
>>> 

[jdtuck]

Thanks, this should be linking against blas, not sure why its not, need to look at Cython arguments. While I am debugging, can you try to install openblas and see if you get the same error?

[echristophe]

Same after installing openblas (and recompiling fdasrsf). I also reproduced the same issue on another clean debian testing machine.

[jdtuck]

Just out of curiosity, does np.__config__.show() show using openblas after you installed it. Usually on Debian you need to do a update-alternatives

[jdtuck]

Can you pull from master and let me know if that fixes it....have only tested on anaconda lately, my bad.

[echristophe]

Thanks for the quick fix!

Few points:

1. your latest change at head fixes my original issue: I do not run into the undefined symbol any more
2. you're right about the openblas, the np.__config__.show() was unchanged (I never ran the update-alternatives)
3. I'm now running into another issue (but I'll need to dig more into it):

   >>> import fdasrsf
   Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/home/echristophe/opensource/fdasrsf_python/fdasrsf/__init__.py", line 29, in <module>
   from .regression import elastic_prediction, elastic_logistic, elastic_regression, elastic_mlogistic
   File "/home/echristophe/opensource/fdasrsf_python/fdasrsf/regression.py", line 14, in <module>
   from patsy import bs
   File "/usr/lib/python3/dist-packages/patsy/__init__.py", line 77, in <module>
   import patsy.highlevel
   File "/usr/lib/python3/dist-packages/patsy/highlevel.py", line 19, in <module>
   from patsy.design_info import DesignMatrix, DesignInfo
   File "/usr/lib/python3/dist-packages/patsy/design_info.py", line 31, in <module>
   from patsy.util import atleast_2d_column_default
   File "/usr/lib/python3/dist-packages/patsy/util.py", line 33, in <module>
   import pandas
   File "/usr/lib/python3/dist-packages/pandas/__init__.py", line 57, in <module>
   from pandas.io.api import *
   File "/usr/lib/python3/dist-packages/pandas/io/api.py", line 19, in <module>
   from pandas.io.packers import read_msgpack, to_msgpack
   File "/usr/lib/python3/dist-packages/pandas/io/packers.py", line 69, in <module>
   from pandas.util._move import (
   ValueError: module functions cannot set METH_CLASS or METH_STATIC

[jdtuck]

This might be related to

https://github.com/pandas-dev/pandas/issues/23040

and its the import order, however I have not seen this on other systems/os