search

ammarhakim / gkyl

This is the main source repo for the Gkeyll 2.0 code. Please see gkeyll.rtfd.io for details.
https://gkeyll.readthedocs.io/en/latest/
54 stars 14 forks source link

Error when compiling: "Undefined reference to 'LAPACKE_dgesv'" #170

Closed skylord-a52 closed 3 months ago

skylord-a52 commented 4 months ago

I'm attempting to install Gkeyll for Ubuntu via WSL. My system is a HP Pavilion Aero Laptop 13-be1xxx, running Windows 11.

I'm able to set up dependencies and run the configuration just fine. When I attempt to compile Gkeyll itself (./waf build install), I get the following error:

[16/24] Creating build/gkylgittip.h
[18/24] Linking build/gkyl
/usr/bin/ld: /home/andrea/gkylsoft/gkylzero/lib/libgkylzero.so: undefined reference to `LAPACKE_dgesv'
collect2: error: ld returned 1 exit status

Waf: Leaving directory `/home/andrea/Sources/gkyl/build'
Build failed
 -> task in 'gkyl' failed with exit status 1 (run with -v to display more information)

I've tried deleting and reinstalling dependencies, as well as building from source instead of from machine files. I always run into this error.

Am I missing some package that I need to apt-get? Is there a config option I'm missing?

JunoRavin commented 4 months ago

This error is suggestive that the LAPACK dependency did not build properly or is not being linked against properly. A couple of suggestions:

  1. In the gkyl/install-deps directory, if the gkylzero library built correctly, you should see a gkylzero directory which should have a Makefile in it. Try running "make check" to see if you can build the unit tests and run those tests (not all of our unit tests currently pass, but there are tests that use LAPACK that will fuss if this is not built correctly).
  2. Check that the configure is indeed finding OpenBLAS. What does your successful configure look like? Does it look like what is in our installation instructions?
  3. Add the path to the OpenBLAS lib like how in our instructions we suggest you may have to add the openmpi lib to the LD_LIBRARY_PATH export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/gkylsoft/openmpi/lib but for the OpenBLAS lib.
skylord-a52 commented 4 months ago
  1. In the gkyl/install-deps directory, if the gkylzero library built correctly, you should see a gkylzero directory which should have a Makefile in it. Try running "make check" to see if you can build the unit tests and run those tests (not all of our unit tests currently pass, but there are tests that use LAPACK that will fuss if this is not built correctly).

The unit tests don't compile for the exact same reason -- liggkylzero is missing LAPACKE.

mkdir -p build/unit
cc -O3 -g -ffast-math -fPIC -MMD -MP   -o build/unit/ctest_alloc unit/ctest_alloc.c -I. -Iminus -Iminus/STC/include -Izero -Iapps -Iregression -Ibuild -Iminus -Iminus/STC -Iminus/STC/include -Iminus/STC/include/stc -Iminus/STC/include/stc/alt -Iminus/STC/docs -Iminus/STC/docs/pics -Izero -Iapps -Ikernels -Ikernels/lbo -Ikernels/dg_diffusion_gen -Ikernels/pkpm -Ikernels/gyrokinetic -Ikernels/vlasov -Ikernels/prim_vars -Ikernels/fem_parproj -Ikernels/euler -Ikernels/dg_diffusion_gyrokinetic -Ikernels/advection -Ikernels/dg_diffusion_fluid -Ikernels/sr_vlasov -Ikernels/maxwell -Ikernels/bin_op -Ikernels/ambi_bolt_potential -Ikernels/dg_diffusion_vlasov -Ikernels/fem_poisson -Ikernels/array_integrate -Ikernels/neutral -Ikernels/basis -Ikernels/fem_poisson_perp -Ikernels/twistshift -Ikernels/fpo -I/home/andrea/gkylsoft/OpenBLAS/include -I/home/andrea/gkylsoft/superlu/include -Izero  -Izero  -Izero   -L/home/andrea/gkylsoft/superlu/lib -L/home/andrea/gkylsoft/OpenBLAS/lib -Lbuild -L. -L. -L.  build/libgkylzero.so -lsuperlu /home/andrea/gkylsoft/OpenBLAS/lib/libopenblas.a     -lm -lpthread -ldl
/usr/bin/ld: build/libgkylzero.so: undefined reference to `LAPACKE_dgesv'
collect2: error: ld returned 1 exit status
make: *** [Makefile:172: build/unit/ctest_alloc] Error 1
  1. Check that the configure is indeed finding OpenBLAS. What does your successful configure look like? Does it look like what is in our installation instructions?
./waf CC=gcc CXX=g++ MPICC=/home/andrea/gkylsoft/openmpi/bin/mpicc MPICXX=/home/andrea/gkylsoft/openmpi/bin/mpicxx --out=build -p /home/andrea/gkylsoft --prefix=/home/andrea/gkylsoft/gkyl --cxxflags=-O3,-std=c++17 --luajit-inc-dir=/home/andrea/gkylsoft/luajit/include/luajit-2.1 --luajit-lib-dir=/home/andrea/gkylsoft/luajit/lib --luajit-share-dir=/home/andrea/gkylsoft/luajit/share/luajit-2.1.0-beta3 --enable-mpi --mpi-inc-dir=/home/andrea/gkylsoft/openmpi/include --mpi-lib-dir=/home/andrea/gkylsoft/openmpi/lib --mpi-link-libs=mpi --enable-adios --adios-inc-dir=/home/andrea/gkylsoft/adios2/include --adios-lib-dir=/home/andrea/gkylsoft/adios2/lib --adios-link-libs=adios2_c_mpi --enable-gkylzero --gkylzero-inc-dir=/home/andrea/gkylsoft/gkylzero/include --gkylzero-lib-dir=/home/andrea/gkylsoft/gkylzero/lib --enable-superlu --superlu-inc-dir=/home/andrea/gkylsoft/superlu/include --superlu-lib-dir=/home/andrea/gkylsoft/superlu/lib --enable-openblas --openblas-inc-dir=/home/andrea/gkylsoft/OpenBLAS/include --openblas-lib-dir=/home/andrea/gkylsoft/OpenBLAS/lib configure
Setting top to                           : /home/andrea/Sources/gkyl
Setting out to                           : /home/andrea/Sources/gkyl/build
Checking for 'gcc' (C compiler)          : gcc
Checking for 'g++' (C++ compiler)        : g++
Setting dependency path:                 : /home/andrea/gkylsoft
Setting prefix:                          : /home/andrea/gkylsoft/gkyl
Checking for LUAJIT                      : Found LuaJIT
Checking for MPI                         : Found MPI
Checking for ADIOS                       : Found ADIOS
Checking for Sqlite3                     : Using Sqlite3
Checking for SUPERLU                     : Found SUPERLU
Checking for OPENBLAS                    : Found OPENBLAS
Checking for gkylzero                    : Found gkylzero
'configure' finished successfully (1.504s)

It looks like it found OPENBLAS just fine. I'm not sure if that means that OPENBLAS is functional though -- it just knows where it is.

  1. Add the path to the OpenBLAS lib like how in our instructions we suggest you may have to add the openmpi lib to the LD_LIBRARY_PATH export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/gkylsoft/openmpi/lib but for the OpenBLAS lib.

I ran export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/gkylsoft/OpenBLAS/lib and that doesn't seem to have changed anything (gkyl and unit tests both fail to compile). Should I try adding it to the general PATH variable?

JunoRavin commented 4 months ago

Apologies, I forgot that lapacke.h header is in the include directory which has the relevant functions. You can try adding the OpenBLAS/include to your $PATH and see if that works. The other zeroth order thing to check is does the OpenBLAS/include directory have lapacke.h in it?

skylord-a52 commented 4 months ago

Adding it to my PATH didn't do anything.

The header file is there, and it does have a definition for LAPACKE_dgesv. It also looks like the only file that uses that function (gkylzero/zero/mat.c) also properly includes it. I guess that means it's an issue with the linker or something? I'm afraid I'm not too familiar with C or C compilation.

JunoRavin commented 4 months ago

Yea the problem is that the shared library is not being linked to the dependency on Lapack even though Lapack is installed when you tell Gkeyll to install the dependencies (such as OpenBLAS, MPI, etc.).

Without knowing how your system is configured, I can't anticipate every issue that might be occurring. A possible solution is to not use the OpenBLAS Gkeyll installs for you when you install the dependencies and instead utilize a system OpenBLAS (for example, if you already have this library from another project, or you want to apt-get OpenBLAS and just have a system version).

To give a precise example, on my Linux workstation I got OpenBLAS just from apt-get and in the configure for gkylzero I have the following lib and includes set:

# Library paths                                                                                                   
LAPACK_INC=/usr/include/
LAPACK_LIB=/usr/lib/x86_64-linux-gnu/liblapacke.a /usr/lib/x86_64-linux-gnu/libopenblas.a /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.a

In my list of installed packages on this Linux workstation I have:

libblas3/jammy,now 3.10.0-2ubuntu1 amd64 [installed,automatic]
liblapack-dev/jammy,now 3.10.0-2ubuntu1 amd64 [installed]
liblapack3/jammy,now 3.10.0-2ubuntu1 amd64 [installed,automatic]
liblapacke-dev/jammy,now 3.10.0-2ubuntu1 amd64 [installed]
liblapacke/jammy,now 3.10.0-2ubuntu1 amd64 [installed]
libopenblas-dev/jammy,now 0.3.20+ds-1 amd64 [installed]
libopenblas-pthread-dev/jammy,now 0.3.20+ds-1 amd64 [installed,automatic]
libopenblas0-pthread/jammy,now 0.3.20+ds-1 amd64 [installed]
libopenblas0/jammy,now 0.3.20+ds-1 amd64 [installed,automatic]
skylord-a52 commented 3 months ago

Well, somehow I fixed it. Not entirely sure how, so sorry if you're reading this and the same thing is happening to you.

Things I did this last time:

Thanks for your help anyway!