search

mpimd-csc / flexiblas

FlexiBLAS - A BLAS and LAPACK wrapper library with runtime exchangeable backends. This is only a mirror of https://gitlab.mpi-magdeburg.mpg.de/software/flexiblas-release
https://www.mpi-magdeburg.mpg.de/projects/flexiblas
GNU Lesser General Public License v3.0
36 stars 7 forks source link

3.4.2: issue with build `flexiblas` binaries #48

Closed kloczek closed 3 months ago

kloczek commented 3 months ago

I've manage to build my own flexiblas binaries and package all that as rpm package. On top of that I've build my own python numpy binaries wiecha are using flexiblas however on any use of those numpy fails with core dump and flexiblas Failed to get the BLAS backend (OPENBLAS-OPENMP) from the configuration. message. Fir example on building joblib python module documentation I have:

+ /usr/bin/sphinx-build -n -T -b man doc build/sphinx/man
Running Sphinx v7.2.6
flexiblas Failed to get the BLAS backend (OPENBLAS-OPENMP) from the configuration.
/var/tmp/rpm-tmp.Dgu6DF: line 59: 893244 Aborted                 (core dumped)

My understanding is that something is missing or even flexiblas is indirectly build however I'm struggling with diagnose this issue and error message does not point and what is missing/wwong.

May I ask for some help/hint? πŸ€”

flexiblas was build using cmake params:

%cmake \
        -D CMAKE_SKIP_INSTALL_RPATH=ON \
        -D INTEGER8=OFF \
        -D SYS_BLAS_LIBRARY=-lblas \
        -D SYS_LAPACK_LIBRARY=-llapack \
        -D TESTS=ON \

and after install flexiblas package flexiblas list shows

[tkloczko@pers-jacek SPECS]$ flexiblas list
System-wide:
System-wide (config directory):
 NETLIB
   library = libflexiblas_netlib.so
   comment =
User config:
Host config:
Enviroment config:

and flexiblas print

[tkloczko@pers-jacek SPECS]$ flexiblas print
FlexiBLAS, version 3.4.2
Copyright (C) 2013-2024 Martin Koehler and others.
This is free software; see the source code for copying conditions.
There is ABSOLUTELY NO WARRANTY; not even for MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE.

Configured BLAS libraries:
System-wide (/etc/flexiblasrc):

System-wide from config directory (/etc/flexiblasrc.d/)
 NETLIB
   library = libflexiblas_netlib.so
   comment =

User config (/home/tkloczko/.flexiblasrc):

Host config (/home/tkloczko/.flexiblasrc.pers-jacek):

Available hooks:

Backend and hook search paths:
  /usr/lib64/flexiblas/

Default BLAS:
    System:       OPENBLAS-OPENMP
    User:         (none)
    Host:         (none)
    Active Default: OPENBLAS-OPENMP (System)
Run-time properties:
   verbose = 0 (System)
Enchufa2 commented 3 months ago

Something against reusing this spec? :)

kloczek commented 3 months ago

Initially I've started my work on top of that spec. Nevertheless what I've produces is not working as expected so I'm looking for how to diagnose this issue πŸ€”

kloczek commented 3 months ago

I've been trying to diagnose this issue using strace -fe trace=file usr/bin/sphinx-build -n -T -b man doc build/sphinx/man and to be hones cannot find anything which may cause this error message

[tkloczko@pers-jacek joblib-1.4.2]$ strace -fe trace=file /usr/bin/sphinx-build -n -T -b man doc build/sphinx/man
execve("/usr/bin/sphinx-build", ["/usr/bin/sphinx-build", "-n", "-T", "-b", "man", "doc", "build/sphinx/man"], 0x7ffe1952bfc0 /* 28 vars */) = 0
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3

[..]

openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/lib64/libflexiblas.so.3", O_RDONLY|O_CLOEXEC) = 4
access("/etc/flexiblasrc", F_OK)        = 0
openat(AT_FDCWD, "/etc/flexiblasrc", O_RDONLY) = 4
readlink("/etc", 0x7ffe50433dc0, 1023)  = -1 EINVAL (Invalid argument)
readlink("/etc/flexiblasrc", 0x7ffe50433dc0, 1023) = -1 EINVAL (Invalid argument)
newfstatat(AT_FDCWD, "/etc/flexiblasrc.d/", {st_mode=S_IFDIR|0755, st_size=22, ...}, 0) = 0
openat(AT_FDCWD, "/etc/flexiblasrc.d/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
openat(AT_FDCWD, "/etc/flexiblasrc.d//netlib.conf", O_RDONLY) = 5
readlink("/etc", 0x7ffe50433dc0, 1023)  = -1 EINVAL (Invalid argument)
readlink("/etc/flexiblasrc.d", 0x7ffe50433dc0, 1023) = -1 EINVAL (Invalid argument)
readlink("/etc/flexiblasrc.d/netlib.conf", 0x7ffe50433dc0, 1023) = -1 EINVAL (Invalid argument)
access("/home/tkloczko/.flexiblasrc", F_OK) = -1 ENOENT (No such file or directory)
access("/home/tkloczko/.flexiblasrc.pers-jacek", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib64/flexiblas/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
access("libflexiblas_netlib.so", R_OK)  = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib64/flexiblas//libflexiblas_netlib.so", {st_mode=S_IFREG|0755, st_size=44248, ...}, 0) = 0
openat(AT_FDCWD, "/usr/lib64/flexiblas//libflexiblas_netlib.so", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/lib64/libblas.so.3", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/lib64/libgfortran.so.5", O_RDONLY|O_CLOEXEC) = 4
access("libflexiblas_fallback_lapack.so", R_OK) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib64/flexiblas//libflexiblas_fallback_lapack.so", {st_mode=S_IFREG|0755, st_size=14952, ...}, 0) = 0
openat(AT_FDCWD, "/usr/lib64/flexiblas//libflexiblas_fallback_lapack.so", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/lib64/liblapack.so.3", O_RDONLY|O_CLOEXEC) = 4
flexiblas Failed to get the BLAS backend (OPENBLAS-OPENMP) from the configuration.
--- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=1081238, si_uid=1000} ---
+++ killed by SIGABRT (core dumped) +++

BTW: why flexiblas is trying to use DSO libflexiblas_fallback_lapack.so from current directory? πŸ€” This may cause some security issues. Is it possible to block that somehow? πŸ€”

grisuthedragon commented 3 months ago

Regarding your flexiblas print output you are trying to load a not known BLAS implementation. OpenBLAS-OpenMP is set in the config but not available. Since you are talking about an own RPM package, did you install all backends?

Can you show the content of /usr/lib64/flexiblas/?

Enchufa2 commented 3 months ago

Even if you don't use my spec, this command may help you get all the build dependencies:

$ sudo dnf builddep flexiblas
kloczek commented 3 months ago

Can you show the content of /usr/lib64/flexiblas/?

[tkloczko@pers-jacek SPECS]$ ls -la /usr/lib64/flexiblas/
total 60
drwxr-xr-x 1 root root   106 May 10 19:41 .
dr-xr-xr-x 1 root root 23990 May 10 19:41 ..
-rwxr-xr-x 1 root root 14952 May 10 00:57 libflexiblas_fallback_lapack.so
-rwxr-xr-x 1 root root 44256 May 10 00:57 libflexiblas_netlib.so

[tkloczko@pers-jacek SPECS]$ flexiblas list
System-wide:
System-wide (config directory):
 NETLIB
   library = libflexiblas_netlib.so
   comment =
User config:
Host config:
Enviroment config:

[tkloczko@pers-jacek SPECS]$ flexiblas print
FlexiBLAS, version 3.4.2
Copyright (C) 2013-2024 Martin Koehler and others.
This is free software; see the source code for copying conditions.
There is ABSOLUTELY NO WARRANTY; not even for MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE.

Configured BLAS libraries:
System-wide (/etc/flexiblasrc):

System-wide from config directory (/etc/flexiblasrc.d/)
 NETLIB
   library = libflexiblas_netlib.so
   comment =

User config (/home/tkloczko/.flexiblasrc):

Host config (/home/tkloczko/.flexiblasrc.pers-jacek):

Available hooks:

Backend and hook search paths:
  /usr/lib64/flexiblas/

Default BLAS:
    System:       OPENBLAS-OPENMP
    User:         (none)
    Host:         (none)
    Active Default: OPENBLAS-OPENMP (System)
Run-time properties:
   verbose = 0 (System)
kloczek commented 3 months ago

Any hints/advices? πŸ€”

grisuthedragon commented 3 months ago

You missing installing the backends. The folder 

ls -la /usr/lib64/flexiblas/
total 60
drwxr-xr-x 1 root root   106 May 10 19:41 .
dr-xr-xr-x 1 root root 23990 May 10 19:41 ..
-rwxr-xr-x 1 root root 14952 May 10 00:57 libflexiblas_fallback_lapack.so
-rwxr-xr-x 1 root root 44256 May 10 00:57 libflexiblas_netlib.so

only contains the netlib implementation and the falllback codes for LAPACK. But in your runtime config you are trying to load OPENBLAS-OPENMP which seem to be not installed.

You have two options...

  1. install FlexiBLAS properly, e.g. by installing sudo dnf builddep flexiblas first
  2. remove OpenBLAS-OPENMP from your system-wide config.
kloczek commented 3 months ago

Issue is that i my case libflexiblas_openblas-openmp.so is not compiled, linked and installed. here is cmake output

+ /usr/bin/cmake -B x86_64-redhat-linux-gnu -D BUILD_SHARED_LIBS=ON -D CMAKE_AR=/usr/bin/gcc-ar -D CMAKE_BUILD_TYPE=RelWithDebInfo -D CMAKE_C_FLAGS_RELEASE=-DNDEBUG -D CMAKE_CXX_FLAGS_RELEASE=-DNDEBUG -D CMAKE_Fortran_FLAGS_RELEASE=-DNDEBUG -D CMAKE_INSTALL_PREFIX=/usr -D CMAKE_NM=/usr/bin/gcc-nm -D CMAKE_RANLIB=/usr/bin/gcc-ranlib -D CMAKE_VERBOSE_MAKEFILE=ON -D INCLUDE_INSTALL_DIR=/usr/include -D LIB_INSTALL_DIR=/usr/lib64 -D LIB_SUFFIX=64 -D SHARE_INSTALL_PREFIX=/usr/share -D SYSCONF_INSTALL_DIR=/etc -S . -D CMAKE_SKIP_INSTALL_RPATH=ON -D INTEGER8=OFF -D LAPACK_API_VERSION=3.4.2 -D SYS_BLAS_LIBRARY=-lblas -D SYS_LAPACK_LIBRARY=-llapack -D TESTS=ON
CMake Warning (dev) at CMakeLists.txt:4 (PROJECT):
  cmake_minimum_required() should be called prior to this top-level project()
  call.  Please see the cmake-commands(7) manual for usage documentation of
  both commands.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- The C compiler identification is GNU 14.1.1
-- The Fortran compiler identification is GNU 14.1.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /usr/bin/gfortran - skipped
-- Configuring FlexiBLAS 3.4.2
-- SYSCONFDIR: /etc
-- LIBDIR:     /usr/lib64
-- BINDIR:     /usr/bin
-- INCLUDEDIR: /usr/include
-- MANDIR:     /usr/share/man
-- Compiler_ID: GNU
-- FCompiler_ID: GNU
-- Fortran compiler supports -frecursive
-- Fortran compiler supports -fPIC
-- Fortran compiler supports -O3
-- Fortran compiler supports -Wimplicit-procedure
-- Fortran compiler supports -Wall
-- Fortran compiler supports -Wunused
-- Fortran compiler supports -Warray-temporaries
-- Fortran compiler supports -fbacktrace
-- Fortran compiler supports -Wshadow
-- Fortran compiler supports -O0
-- Performing Test C_W_PEDANTIC
-- Performing Test C_W_PEDANTIC - Success
-- Performing Test C_W_STRICT_PROTOTYPES
-- Performing Test C_W_STRICT_PROTOTYPES - Success
-- Performing Test C_W_CAST_QUAL
-- Performing Test C_W_CAST_QUAL - Success
-- Performing Test C_STACK_PROTECTOR
-- Performing Test C_STACK_PROTECTOR - Success
-- Performing Test C_STACK_CLASH
-- Performing Test C_STACK_CLASH - Success
-- Performing Test FC_STACK_PROTECTOR
-- Performing Test FC_STACK_PROTECTOR - Success
-- Performing Test FC_STACK_CLASH
-- Performing Test FC_STACK_CLASH - Success
-- Detecting Fortran/C Interface
-- Detecting Fortran/C Interface - Found GLOBAL and MODULE mangling
-- Verifying Fortran/C Compiler Compatibility
-- Verifying Fortran/C Compiler Compatibility - Success
-- Check for Thread Local Storage
-- Performing Test HAVE_THREAD_LOCAL
-- Performing Test HAVE_THREAD_LOCAL - Success
-- Performing Test HAVE__THREAD_LOCAL
-- Performing Test HAVE__THREAD_LOCAL - Success
-- Performing Test HAVE_GNU_THREAD_LOCAL
-- Performing Test HAVE_GNU_THREAD_LOCAL - Success
-- Looking for include file dlfcn.h
-- Looking for include file dlfcn.h - found
-- Looking for include file sys/sysmacro.h
-- Looking for include file sys/sysmacro.h - not found
-- Looking for include file sys/stat.h
-- Looking for include file sys/stat.h - found
-- Looking for RTLD_GLOBAL
-- Looking for RTLD_GLOBAL - found
-- Looking for RTLD_DEEPBIND
-- Looking for RTLD_DEEPBIND - found
-- Looking for RTLD_NOW
-- Looking for RTLD_NOW - found
-- Looking for RTLD_NODELETE
-- Looking for RTLD_NODELETE - found
-- Looking for dlopen
-- Looking for dlopen - found
-- Looking for dlopen
-- Looking for dlopen - found
-- Check for __attribute__((constructor))
-- Looking for attribute:  constructor  - works
-- Check for __attribute__((alias("foo")))
-- Looking for attribute:  alias("foo")  - works
-- Check for __attribute__((visibility("hidden")))
-- Looking for attribute:  visibility("hidden")  - works
-- Check for __attribute__((visibility("default")))
-- Looking for attribute:  visibility("default")  - works
-- Check for __attribute__((unused))
-- Looking for attribute:  unused  - works
-- libcscutils in project build

-- LIBCSCUTILS Features
-- Enable module: inifile
-- Enable module: ds

-- Check Generic Functionality
-- Check for __attribute__((constructor))
-- Looking for attribute:  constructor  - works
-- Check for __attribute__((deprecated))
-- Looking for attribute:  deprecated  - works
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Looking for pthread_yield
-- Looking for pthread_yield
-- Looking for pthread_yield - not found
-- Looking for pthread_yield
-- Looking for pthread_yield - not found
-- Looking for pthread_yield
-- Looking for pthread_yield - not found
-- Looking for pthread_yield
-- Looking for pthread_yield - not found
-- Looking for pthread_yield
-- Looking for pthread_yield - not found
-- Looking for pthread_yield
-- Looking for pthread_yield - not found
-- Looking for pthread_yield
-- Looking for pthread_yield - not found
-- Looking for pthread_yield - not found.
-- Looking for include file execinfo.h
-- Looking for include file execinfo.h - found
-- Looking for backtrace
-- Looking for backtrace - found
-- Looking for backtrace_symbols
-- Looking for backtrace_symbols - found
-- Enable csc_show_backtrace.
-- Looking for strnlen
-- Looking for strnlen
-- Looking for strnlen - not found
-- Looking for strnlen
-- Looking for strnlen - not found
-- Looking for strnlen
-- Looking for strnlen - not found
-- Looking for strnlen
-- Looking for strnlen - found
-- Looking for strnlen - found (using _POSIX_C_SOURCE=200809L).
-- Looking for strndup
-- Looking for strndup
-- Looking for strndup - not found
-- Looking for strndup
-- Looking for strndup - not found
-- Looking for strndup
-- Looking for strndup - not found
-- Looking for strndup
-- Looking for strndup - found
-- Looking for strndup - found (using _POSIX_C_SOURCE=200809L).
-- Looking for include file sys/ioctl.h
-- Looking for include file sys/ioctl.h - found
-- Looking for include file unistd.h
-- Looking for include file unistd.h - found
-- Looking for getline
-- Looking for getline
-- Looking for getline - not found
-- Looking for getline
-- Looking for getline - not found
-- Looking for getline
-- Looking for getline - not found
-- Looking for getline
-- Looking for getline - found
-- Looking for getline - found (using _POSIX_C_SOURCE=200809L).
-- Looking for getdelim
-- Looking for getdelim
-- Looking for getdelim - not found
-- Looking for getdelim
-- Looking for getdelim - not found
-- Looking for getdelim
-- Looking for getdelim - not found
-- Looking for getdelim
-- Looking for getdelim - found
-- Looking for getdelim - found (using _POSIX_C_SOURCE=200809L).

-- Features for CSC-INIFILE
-- Looking for realpath
-- Looking for realpath
-- Looking for realpath - found
-- Looking for realpath - found (using _NONE_).

-- External Libraries
-- Could NOT find CPUFREQ (missing: CPUFREQ_LIBRARIES CPUFREQ_INCLUDE_DIR)
CPUFREQ not found. Either cpufreq.h or libcpupower.so is missing.
-- Looking for strdup
-- Looking for strdup
-- Looking for strdup - not found
-- Looking for strdup
-- Looking for strdup - not found
-- Looking for strdup
-- Looking for strdup - not found
-- Looking for strdup
-- Looking for strdup - found
-- Looking for strdup - found (using _POSIX_C_SOURCE=200809L).
-- Looking for realpath
-- Looking for realpath
-- Looking for realpath - found
-- Looking for realpath - found (using _NONE_).
-- Use external LAPACK from -llapack with API_VERSION 3.4.2
CMake Warning at cmake/lapack.cmake:242 (MESSAGE):
  The requested LAPACK_VERSION is less than 3.6.1.  We recommend to use a
  newer API version.
Call Stack (most recent call first):
  CMakeLists.txt:352 (INCLUDE)

-- Use system provided NETLIB BLAS for fallback generation. SYS_BLAS_LIBRARY = -lblas
-- Use system provided LAPACK for fallback generation. SYS_LAPACK_LIBRARY = -llapack
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_Fortran: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- OpenBLAS-Serial:  Candidate: /usr/lib64/libopenblas.so
-- OpenBLAS-Serial:  /usr/lib64/libopenblas.so does not contain the Serial version of OpenBLAS.
-- Could NOT find OpenBLASSerial (missing: OpenBLASSerial_LIBRARY)
-- OpenBLASSerial not found.
-- OpenBLAS-PThread:  Candidate: /usr/lib64/libopenblas.so
-- Found OpenBLASPThread: /usr/lib64/libopenblas.so
-- Build BLAS interface: OpenBLASPThread
-- Libs: /usr/lib64/libopenblas.so
-- Build additional BLAS interface: OpenBLASPThread
-- -- Libs: /usr/lib64/libopenblas.so
-- -- Linker Flags:
-- Performing Test ZDOTC_MKL
-- Performing Test ZDOTC_MKL - Failed
-- Looking for flexiblas_exit
-- Looking for flexiblas_exit - not found
-- Looking for Fortran DAXPY
-- Looking for Fortran DAXPY - found
-- Looking for cblas_daxpy
-- Looking for cblas_daxpy - found
-- Looking for MKL_Get_Version
-- Looking for MKL_Get_Version - not found
-- Add CBLAS in dummy library
-- OpenBLAS-OpenMP:  Candidate: /usr/lib64/libopenblas.so
-- OpenBLAS-OpenMP:  /usr/lib64/libopenblas.so does not contain the OpenMP version of OpenBLAS.
-- Could NOT find OpenBLASOpenMP (missing: OpenBLASOpenMP_LIBRARY)
-- OpenBLASOpenMP not found.
-- Could NOT find BlisSerial (missing: BlisSerial_LIBRARY)
-- BlisSerial not found.
-- Could NOT find BlisPThread (missing: BlisPThread_LIBRARY)
-- BlisPThread not found.
-- Could NOT find BlisOpenMP (missing: BlisOpenMP_LIBRARY)
-- BlisOpenMP not found.
-- Search for MKL: mkl_gf_lp64 mkl_sequential mkl_core
-- MklSerial not found.
-- Search for MKL: mkl_gf_lp64 mkl_gnu_thread mkl_core
-- MklOpenMP not found.
-- PP:
-- SP:
-- MKL_TBB depends on TBB. Not Found.
-- MklTBB not found.
--
-- Search for ATLAS
-- A library with BLAS API not found. Please specify library location.
--
-- Search for Apple
-- A library with BLAS API not found. Please specify library location.
-- Default BLAS  not found. Use NETLIB instead.
-- Found hook directory.
-- Build profile hook.
-- Enable Multithread Demo
-- Build Extra Benchmarks
-- OpenBLASPThread - /usr/lib64/libopenblas.so -
-- Found PythonInterp: /usr/bin/python3
-- Found Python (3.10.14): /usr/include/python3.10
-- Enable Testing...
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/sblat1.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/sblat2.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/sblat3.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/dblat1.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/dblat2.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/dblat3.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/cblat1.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/cblat2.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/cblat3.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/zblat1.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/zblat2.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/zblat3.in
-- ------------------------------------------------
--       Final Configuration Status
-- ------------------------------------------------
-- Build 8 byte integer: OFF
-- Build CBLAS interface: ON
-- Install path: /usr
-- Extra BLAS search path: /usr/lib64/flexiblas/
-- LAPACK Support: ON
-- LAPACK_API_VERSION: 3.4.2
-- ABI: GNU
-- Interface to Extra BLAS Libraries:
--  /usr/lib64/libopenblas.so - OpenBLASPThread
-- Default BLAS: NETLIB
--
-- Compiler Options:
--  CMAKE_C_FLAGS               = -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O2 -g -grecord-gcc-switches -pipe -mtls-dialect=gnu2 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fdata-sections -ffunction-sections -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -flto=auto -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -Wall -Werror=format-security -fPIC -std=c99 -fstack-protector-strong -fstack-clash-protection -D_FILE_OFFSET_BITS=64
--  CMAKE_C_FLAGS_RELEASE       = -DNDEBUG -O3 -Wpedantic -Wstrict-prototypes -Wcast-qual
--  CMAKE_C_FLAGS_DEBUG         = -g -Wall -Wunused -g -Wextra -Wshadow -Wstrict-aliasing -Wundef -Wlogical-op -Wparentheses -Wcast-qual -O0 -Wpedantic -Wstrict-prototypes -Wcast-qual
--  CMAKE_Fortran_FLAGS         = -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O2 -g -grecord-gcc-switches -pipe -mtls-dialect=gnu2 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fdata-sections -ffunction-sections -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -flto=auto -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -Wall -Werror=format-security -I/usr/lib64/gfortran/modules -frecursive -fPIC -fstack-protector-strong -fstack-clash-protection
--  CMAKE_Fortran_FLAGS_RELEASE = -DNDEBUG -O3
--  CMAKE_Fortran_FLAGS_DEBUG   = -g -Wimplicit-procedure -Wall -Wunused -Warray-temporaries -fbacktrace -Wshadow -O0
-- ------------------------------------------------
-- Configuring done (23.3s)
-- Generating done (0.5s)
CMake Warning:
  Manually-specified variables were not used by the project:

    INCLUDE_INSTALL_DIR
    LIB_INSTALL_DIR
    LIB_SUFFIX
    SHARE_INSTALL_PREFIX
    SYSCONF_INSTALL_DIR

I'm building temporary flexiblas without blis and atlas installed.

kloczek commented 3 months ago

Above cmake produces only belom list of loadable modules

-rwxr-xr-x 1 tkloczko tkloczko  14952 May 13 11:56 libflexiblas_fallback_lapack.so
-rwxr-xr-x 1 tkloczko tkloczko  15688 May 13 11:56 libflexiblas_hook_dummy.so
-rwxr-xr-x 1 tkloczko tkloczko 819336 May 13 11:56 libflexiblas_hook_profile.so
-rwxr-xr-x 1 tkloczko tkloczko  44256 May 13 11:56 libflexiblas_netlib.so
-rwxr-xr-x 1 tkloczko tkloczko  39744 May 13 11:56 libflexiblas_openblas-threads.so
grisuthedragon commented 3 months ago

Now, setting the default BLAS to OpenBLASPThread should enable to run everything with OpenBLAS.

kloczek commented 3 months ago

But it still do not explain why openblas-openmp backend is not build.

grisuthedragon commented 3 months ago

The log shows 

-- OpenBLAS-OpenMP:  Candidate: /usr/lib64/libopenblas.so
-- OpenBLAS-OpenMP:  /usr/lib64/libopenblas.so does not contain the OpenMP version of OpenBLAS.

so the OpenBLAS version installed on your system is built with Pthreads support and not with OpenMP support.

kloczek commented 3 months ago

I don't see anything about OpenMP support in OpenBLAS 0.3.27 cmake output

+ /usr/bin/cmake -B x86_64-redhat-linux-gnu -D BUILD_SHARED_LIBS=ON -D CMAKE_AR=/usr/bin/gcc-ar -D CMAKE_BUILD_TYPE=RelWithDebInfo -D CMAKE_C_FLAGS_RELEASE=-DNDEBUG -D CMAKE_CXX_FLAGS_RELEASE=-DNDEBUG -D CMAKE_Fortran_FLAGS_RELEASE=-DNDEBUG -D CMAKE_INSTALL_PREFIX=/usr -D CMAKE_NM=/usr/bin/gcc-nm -D CMAKE_RANLIB=/usr/bin/gcc-ranlib -D CMAKE_VERBOSE_MAKEFILE=ON -D INCLUDE_INSTALL_DIR=/usr/include -D LIB_INSTALL_DIR=/usr/lib64 -D LIB_SUFFIX=64 -D SHARE_INSTALL_PREFIX=/usr/share -D SYSCONF_INSTALL_DIR=/etc -S . -D BUILD_STATIC_LIBS=OFF -D BUILD_TESTING=ON -D BUILD_WITHOUT_CBLAS=OFF -D BUILD_WITHOUT_LAPACK=OFF -D CPP_THREAD_SAFETY_GEMV=ON -D CPP_THREAD_SAFETY_TEST=ON -D DYNAMIC_ARCH=OFF -D INTERFACE64=OFF -D USE_LOCKING=OFF -D USE_PERL=ON
CMake Deprecation Warning at CMakeLists.txt:5 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

-- The C compiler identification is GNU 14.1.1
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/gcc
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
CMake Warning at CMakeLists.txt:101 (message):
  CMake support is experimental.  It does not yet support all build options
  and may not produce the same Makefiles that OpenBLAS ships with.

CMake Warning (dev) at cmake/system_check.cmake:13 (if):
  Policy CMP0054 is not set: Only interpret if() arguments as variables or
  keywords when unquoted.  Run "cmake --help-policy CMP0054" for policy
  details.  Use the cmake_policy command to set the policy and suppress this
  warning.

  Quoted variables like "LINUX" will no longer be dereferenced when the
  policy is set to NEW.  Since the policy is not set the OLD behavior will be
  used.
Call Stack (most recent call first):
  cmake/system.cmake:8 (include)
  CMakeLists.txt:104 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Looking for stdatomic.h
-- Looking for stdatomic.h - found
-- GEMM multithread threshold set to 4.
-- Multi-threading enabled with 48 threads.
-- Looking for a Fortran compiler
-- Looking for a Fortran compiler - /usr/bin/gfortran
-- The Fortran compiler identification is GNU 14.1.1
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /usr/bin/gfortran - skipped
-- Running getarch
-- GETARCH results:
CORE=SKYLAKEX
LIBCORE=skylakex
NUM_CORES=48
HAVE_MMX=1
HAVE_SSE=1
HAVE_SSE2=1
HAVE_SSE3=1
HAVE_SSSE3=1
HAVE_SSE4_1=1
HAVE_SSE4_2=1
HAVE_AVX=1
HAVE_AVX2=1
HAVE_AVX512VL=1
HAVE_FMA3=1
MAKEFLAGS += -j 48

-- Compiling a 64-bit binary.
-- Building Single Precision
-- Building Double Precision
-- Building Complex Precision
-- Building Double Complex Precision
-- /home/tkloczko/rpmbuild/BUILD/OpenBLAS-0.3.27/x86_64-redhat-linux-gnu/interface/CMakeFiles/scopy.c

[..]

-- /home/tkloczko/rpmbuild/BUILD/OpenBLAS-0.3.27/x86_64-redhat-linux-gnu/lapack/CMakeFiles/ztrti2_LU.c
-- fortran lapack
-- Building deprecated routines
-- Building Single Precision
-- Building Double Precision
-- Building Single Precision Complex
-- Building Double Precision Complex
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- The CXX compiler identification is GNU 14.1.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Generating openblas_config.h in include/openblas
-- Generating f77blas.h in include/openblas
-- Generating cblas.h in include/openblas
-- Copying LAPACKE header files to include/openblas
-- Configuring done (6.7s)
-- Generating done (1.3s)
grisuthedragon commented 3 months ago

Try compiling OpenBLAS with the standard makefile, e.g. 

make USE_OPENMP=1
Enchufa2 commented 3 months ago

Again, if you want to build openblas yourself, you can use Fedora's openblas spec for reference, as we build the three versions.

kloczek commented 3 months ago

Try compiling OpenBLAS with the standard makefile, e.g.

make USE_OPENMP=1

So does it mean that OpenBLAS cmake is broken? πŸ€”

Enchufa2 commented 3 months ago

What do you mean broken? Did you check the documentation? OpenBLAS provides three flavors. The default one uses pthreads. If that's not what you want, you need to change the proper options.

You have 1) a working OpenBLAS RPM that provides these three flavors in Fedora, I provided the command to install everything you need; 2) a working OpenBLAS spec that builds the latter if you want to do it yourself; 3) a working FlexiBLAS spec that demonstrates how to build it for all backends. Please take a look at these resources and read the documentation. We tried to tell you multiple times.

So far, I don't see any issue with FlexiBLAS: you provided a library, FlexiBLAS found it and is ready to use it. So I think this can be closed. If OpenBLAS documentation + all the examples available are not clear enough, I suggest you to ask for help there.

kloczek commented 3 months ago

What do you mean broken? Did you check the documentation? OpenBLAS provides three flavors. The default one uses pthreads. If that's not what you want, you need to change the proper options.

You have 1) a working OpenBLAS RPM that provides these three flavors in Fedora, I provided the command to install everything you need; 2) a working OpenBLAS spec that builds the latter if you want to do it yourself; 3) a working FlexiBLAS spec that demonstrates how to build it for all backends. Please take a look at these resources and read the documentation. We tried to tell you multiple times.

Checked OpenBLAS documentation and NONE of those options are documented in case of use cmake to build it. Looks like -D USE_OPENMP=ON option is handled in cmake/fc.cmake but even after specify that option it is not listed by cmake -L

Enchufa2 commented 3 months ago

For instance, docs/build_system.md:

USE_THREAD      - Use a multithreading backend (default to pthread)
USE_LOCKING     - implement locking for thread safety even when USE_THREAD is not set (so that the singlethreaded library can
                  safely be called from multithreaded programs)
USE_OPENMP      - Use OpenMP as multithreading backend

among other things. But yet again, we are discussing OpenBLAS... Β―_(ツ)_/Β―

kloczek commented 3 months ago

Thank you for the hints πŸ‘ πŸ˜„

As I've resolved building flexiblas OpenBLAS DSO just found that OpenBLAS is build with:

%build
%cmake \
        -D BUILD_STATIC_LIBS=OFF \
        -D BUILD_TESTING=ON \
        -D BUILD_WITHOUT_CBLAS=OFF \
        -D BUILD_WITHOUT_LAPACK=OFF \
        -D CPP_THREAD_SAFETY_GEMV=ON \
        -D CPP_THREAD_SAFETY_TEST=ON \
        -D DYNAMIC_ARCH=OFF \
        -D INTERFACE64=OFF \
        -D USE_LOCKING=OFF \
        -D USE_OPENMP=ON \
        -D USE_PERL=ON \
        -D USE_THREAD=ON \
        %{nil}
%cmake_build

%if %{with 8bytes_int}
%cmake \
        -B %{_vpath_builddir}-8bytes_int \
        -D BUILD_STATIC_LIBS=OFF \
        -D BUILD_TESTING=ON \
        -D BUILD_WITHOUT_CBLAS=OFF \
        -D BUILD_WITHOUT_LAPACK=OFF \
        -D CPP_THREAD_SAFETY_GEMV=ON \
        -D CPP_THREAD_SAFETY_TEST=ON \
        -D DYNAMIC_ARCH=OFF \
        -D INTERFACE64=ON \
        -D USE_LOCKING=OFF \
        -D USE_OPENMP=ON \
        -D USE_PERL=ON \
        -D USE_THREAD=ON \
        %{nil}
%cmake_build
%endif

On building 64bits int flexiblas OpenBLAS DSO is not build

+ /usr/bin/cmake -B x86_64-redhat-linux-gnu -D BUILD_SHARED_LIBS=ON -D CMAKE_AR=/usr/bin/gcc-ar -D CMAKE_BUILD_TYPE=RelWithDebInfo -D CMAKE_C_FLAGS_RELEASE=-DNDEBUG -D CMAKE_CXX_FLAGS_RELEASE=-DNDEBUG -D CMAKE_Fortran_FLAGS_RELEASE=-DNDEBUG -D CMAKE_INSTALL_PREFIX=/usr -D CMAKE_NM=/usr/bin/gcc-nm -D CMAKE_RANLIB=/usr/bin/gcc-ranlib -D CMAKE_VERBOSE_MAKEFILE=ON -D INCLUDE_INSTALL_DIR=/usr/include -D LIB_INSTALL_DIR=/usr/lib64 -D LIB_SUFFIX=64 -D SHARE_INSTALL_PREFIX=/usr/share -D SYSCONF_INSTALL_DIR=/etc -S . -B x86_64-redhat-linux-gnu-8bytes_int -D CMAKE_SKIP_INSTALL_RPATH=ON -D FLEXIBLAS_DEFAULT_BLAS=ON -D INTEGER8=ON -D LAPACK_API_VERSION=3.4.2 -D SYS_BLAS_LIBRARY=-lblas64 -D SYS_LAPACK_LIBRARY=-llapack -D TESTS=ON
CMake Warning (dev) at CMakeLists.txt:4 (PROJECT):
  cmake_minimum_required() should be called prior to this top-level project()
  call.  Please see the cmake-commands(7) manual for usage documentation of
  both commands.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- The C compiler identification is GNU 14.1.1
-- The Fortran compiler identification is GNU 14.1.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /usr/bin/gfortran - skipped
-- Configuring FlexiBLAS 3.4.2
-- SYSCONFDIR: /etc
-- LIBDIR:     /usr/lib64
-- BINDIR:     /usr/bin
-- INCLUDEDIR: /usr/include
-- MANDIR:     /usr/share/man
-- Compiler_ID: GNU
-- FCompiler_ID: GNU
-- Fortran compiler supports -frecursive
-- Fortran compiler supports -fPIC
-- Fortran compiler supports -O3
-- Fortran compiler supports -Wimplicit-procedure
-- Fortran compiler supports -Wall
-- Fortran compiler supports -Wunused
-- Fortran compiler supports -Warray-temporaries
-- Fortran compiler supports -fbacktrace
-- Fortran compiler supports -Wshadow
-- Fortran compiler supports -O0
-- Performing Test C_W_PEDANTIC
-- Performing Test C_W_PEDANTIC - Success
-- Performing Test C_W_STRICT_PROTOTYPES
-- Performing Test C_W_STRICT_PROTOTYPES - Success
-- Performing Test C_W_CAST_QUAL
-- Performing Test C_W_CAST_QUAL - Success
-- Performing Test C_STACK_PROTECTOR
-- Performing Test C_STACK_PROTECTOR - Success
-- Performing Test C_STACK_CLASH
-- Performing Test C_STACK_CLASH - Success
-- Performing Test FC_STACK_PROTECTOR
-- Performing Test FC_STACK_PROTECTOR - Success
-- Performing Test FC_STACK_CLASH
-- Performing Test FC_STACK_CLASH - Success
-- Detecting Fortran/C Interface
-- Detecting Fortran/C Interface - Found GLOBAL and MODULE mangling
-- Verifying Fortran/C Compiler Compatibility
-- Verifying Fortran/C Compiler Compatibility - Success
-- Check for Thread Local Storage
-- Performing Test HAVE_THREAD_LOCAL
-- Performing Test HAVE_THREAD_LOCAL - Success
-- Performing Test HAVE__THREAD_LOCAL
-- Performing Test HAVE__THREAD_LOCAL - Success
-- Performing Test HAVE_GNU_THREAD_LOCAL
-- Performing Test HAVE_GNU_THREAD_LOCAL - Success
-- Looking for include file dlfcn.h
-- Looking for include file dlfcn.h - found
-- Looking for include file sys/sysmacro.h
-- Looking for include file sys/sysmacro.h - not found
-- Looking for include file sys/stat.h
-- Looking for include file sys/stat.h - found
-- Looking for RTLD_GLOBAL
-- Looking for RTLD_GLOBAL - found
-- Looking for RTLD_DEEPBIND
-- Looking for RTLD_DEEPBIND - found
-- Looking for RTLD_NOW
-- Looking for RTLD_NOW - found
-- Looking for RTLD_NODELETE
-- Looking for RTLD_NODELETE - found
-- Looking for dlopen
-- Looking for dlopen - found
-- Looking for dlopen
-- Looking for dlopen - found
-- Check for __attribute__((constructor))
-- Looking for attribute:  constructor  - works
-- Check for __attribute__((alias("foo")))
-- Looking for attribute:  alias("foo")  - works
-- Check for __attribute__((visibility("hidden")))
-- Looking for attribute:  visibility("hidden")  - works
-- Check for __attribute__((visibility("default")))
-- Looking for attribute:  visibility("default")  - works
-- Check for __attribute__((unused))
-- Looking for attribute:  unused  - works
-- libcscutils in project build

-- LIBCSCUTILS Features
-- Enable module: inifile
-- Enable module: ds

-- Check Generic Functionality
-- Check for __attribute__((constructor))
-- Looking for attribute:  constructor  - works
-- Check for __attribute__((deprecated))
-- Looking for attribute:  deprecated  - works
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Looking for pthread_yield
-- Looking for pthread_yield
-- Looking for pthread_yield - not found
-- Looking for pthread_yield
-- Looking for pthread_yield - not found
-- Looking for pthread_yield
-- Looking for pthread_yield - not found
-- Looking for pthread_yield
-- Looking for pthread_yield - not found
-- Looking for pthread_yield
-- Looking for pthread_yield - not found
-- Looking for pthread_yield
-- Looking for pthread_yield - not found
-- Looking for pthread_yield
-- Looking for pthread_yield - not found
-- Looking for pthread_yield - not found.
-- Looking for include file execinfo.h
-- Looking for include file execinfo.h - found
-- Looking for backtrace
-- Looking for backtrace - found
-- Looking for backtrace_symbols
-- Looking for backtrace_symbols - found
-- Enable csc_show_backtrace.
-- Looking for strnlen
-- Looking for strnlen
-- Looking for strnlen - not found
-- Looking for strnlen
-- Looking for strnlen - not found
-- Looking for strnlen
-- Looking for strnlen - not found
-- Looking for strnlen
-- Looking for strnlen - found
-- Looking for strnlen - found (using _POSIX_C_SOURCE=200809L).
-- Looking for strndup
-- Looking for strndup
-- Looking for strndup - not found
-- Looking for strndup
-- Looking for strndup - not found
-- Looking for strndup
-- Looking for strndup - not found
-- Looking for strndup
-- Looking for strndup - found
-- Looking for strndup - found (using _POSIX_C_SOURCE=200809L).
-- Looking for include file sys/ioctl.h
-- Looking for include file sys/ioctl.h - found
-- Looking for include file unistd.h
-- Looking for include file unistd.h - found
-- Looking for getline
-- Looking for getline
-- Looking for getline - not found
-- Looking for getline
-- Looking for getline - not found
-- Looking for getline
-- Looking for getline - not found
-- Looking for getline
-- Looking for getline - found
-- Looking for getline - found (using _POSIX_C_SOURCE=200809L).
-- Looking for getdelim
-- Looking for getdelim
-- Looking for getdelim - not found
-- Looking for getdelim
-- Looking for getdelim - not found
-- Looking for getdelim
-- Looking for getdelim - not found
-- Looking for getdelim
-- Looking for getdelim - found
-- Looking for getdelim - found (using _POSIX_C_SOURCE=200809L).

-- Features for CSC-INIFILE
-- Looking for realpath
-- Looking for realpath
-- Looking for realpath - found
-- Looking for realpath - found (using _NONE_).

-- External Libraries
-- Could NOT find CPUFREQ (missing: CPUFREQ_LIBRARIES CPUFREQ_INCLUDE_DIR)
CPUFREQ not found. Either cpufreq.h or libcpupower.so is missing.
-- Setting GNU Compiler Flags
-- Looking for strdup
-- Looking for strdup
-- Looking for strdup - not found
-- Looking for strdup
-- Looking for strdup - not found
-- Looking for strdup
-- Looking for strdup - not found
-- Looking for strdup
-- Looking for strdup - found
-- Looking for strdup - found (using _POSIX_C_SOURCE=200809L).
-- Looking for realpath
-- Looking for realpath
-- Looking for realpath - found
-- Looking for realpath - found (using _NONE_).
-- Use external LAPACK from -llapack with API_VERSION 3.4.2
CMake Warning at cmake/lapack.cmake:242 (MESSAGE):
  The requested LAPACK_VERSION is less than 3.6.1.  We recommend to use a
  newer API version.
Call Stack (most recent call first):
  CMakeLists.txt:352 (INCLUDE)

-- Use system provided NETLIB BLAS for fallback generation. SYS_BLAS_LIBRARY = -lblas64
-- Use system provided LAPACK for fallback generation. SYS_LAPACK_LIBRARY = -llapack
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_Fortran: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- Could NOT find OpenBLASSerial64 (missing: OpenBLASSerial64_LIBRARY)
-- OpenBLASSerial64 not found.
-- Could NOT find OpenBLASPThread64 (missing: OpenBLASPThread64_LIBRARY)
-- OpenBLASPThread64 not found.
-- Could NOT find OpenBLASOpenMP64 (missing: OpenBLASOpenMP64_LIBRARY)
-- OpenBLASOpenMP64 not found.
-- Could NOT find BlisSerial64 (missing: BlisSerial64_LIBRARY)
-- BlisSerial64 not found.
-- Could NOT find BlisPThread64 (missing: BlisPThread64_LIBRARY)
-- BlisPThread64 not found.
-- Could NOT find BlisOpenMP64 (missing: BlisOpenMP64_LIBRARY)
-- BlisOpenMP64 not found.
-- Search for MKL: mkl_gf_ilp64 mkl_sequential mkl_core
-- MklSerial64 not found.
-- Search for MKL: mkl_gf_ilp64 mkl_gnu_thread mkl_core
-- MklOpenMP64 not found.
-- PP:
-- SP:
-- MKL_TBB depends on TBB. Not Found.
-- MklTBB64 not found.
--
-- Search for ATLAS
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - found
-- Check for BLAS64 interface: 0 TRUE
-- BLAS64 Library Generic works with 64bit integers
-- Found BLAS64: Generic (/usr/lib64/libblas64.so)
--
-- Search for Apple
-- Check for BLAS64 interface: 0 TRUE
-- BLAS64 Library Generic works with 64bit integers
-- Found BLAS64: Generic (/usr/lib64/libblas64.so)
-- Default BLAS  not found. Use NETLIB instead.
-- Found hook directory.
-- Build profile hook.
-- Enable Multithread Demo
-- Build Extra Benchmarks
-- Found PythonInterp: /usr/bin/python3
-- Found Python (3.10.14): /usr/include/python3.10
-- Enable Testing...
-- Setting GNU Compiler Flags
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/sblat1.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/sblat2.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/sblat3.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/dblat1.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/dblat2.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/dblat3.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/cblat1.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/cblat2.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/cblat3.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/zblat1.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/zblat2.in
-- /home/tkloczko/rpmbuild/BUILD/flexiblas-3.4.2/test/blas/zblat3.in
-- ------------------------------------------------
--       Final Configuration Status
-- ------------------------------------------------
-- Build 8 byte integer: ON
-- Build CBLAS interface: ON
-- Install path: /usr
-- Extra BLAS search path: /usr/lib64/flexiblas64/
-- LAPACK Support: ON
-- LAPACK_API_VERSION: 3.4.2
-- ABI: GNU
-- Interface to Extra BLAS Libraries:
-- Default BLAS: NETLIB
--
-- Compiler Options:
--  CMAKE_C_FLAGS               = -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O2 -g -grecord-gcc-switches -pipe -mtls-dialect=gnu2 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fdata-sections -ffunction-sections -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -flto=auto -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -Wall -Werror=format-security -fPIC -std=c99 -fstack-protector-strong -fstack-clash-protection -D_FILE_OFFSET_BITS=64
--  CMAKE_C_FLAGS_RELEASE       = -DNDEBUG -O3 -Wpedantic -Wstrict-prototypes -Wcast-qual
--  CMAKE_C_FLAGS_DEBUG         = -g -Wall -Wunused -g -Wextra -Wshadow -Wstrict-aliasing -Wundef -Wlogical-op -Wparentheses -Wcast-qual -O0 -Wpedantic -Wstrict-prototypes -Wcast-qual
--  CMAKE_Fortran_FLAGS         = -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -O2 -g -grecord-gcc-switches -pipe -mtls-dialect=gnu2 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fdata-sections -ffunction-sections -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -flto=auto -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -Wall -Werror=format-security -I/usr/lib64/gfortran/modules -frecursive -fPIC -fdefault-integer-8 -fstack-protector-strong -fstack-clash-protection -fdefault-integer-8
--  CMAKE_Fortran_FLAGS_RELEASE = -DNDEBUG -O3
--  CMAKE_Fortran_FLAGS_DEBUG   = -g -Wimplicit-procedure -Wall -Wunused -Warray-temporaries -fbacktrace -Wshadow -O0
-- ------------------------------------------------
-- Configuring done (20.3s)
-- Generating done (0.5s)

Above output is almost identical with that one INTEGER8=OFF.

Any hint? πŸ€”

grisuthedragon commented 3 months ago

If Flexiblas with -DINTEGER8=ON searches for OpenBLAS with OpenMP, the following libraries are checked:

If the library does not have any of the names above, it is not found.

A question in general, what about using the provided spec files or packages @Enchufa2 mentioned? They are well configured and working properly.

kloczek commented 3 months ago

Only now I found in cmake params copied here is -D SYS_BLAS_LIBRARY=-lblas64 -D SYS_LAPACK_LIBRARY=-llapack. This is result of

%if %{with 8bytes_int}
%cmake \
        -B %{_vpath_builddir}-8bytes_int \
        -D CMAKE_SKIP_INSTALL_RPATH=ON \
        -D FLEXIBLAS_DEFAULT_BLAS=ON \
        -D INTEGER8=ON \
        -D LAPACK_API_VERSION="$(rpm -q --qf '%{VERSION}' lapack-devel)" \
        -D SYS_BLAS_LIBRARY="$(pkg-config --libs blas64)" \
        -D SYS_LAPACK_LIBRARY="$(pkg-config --libs lapack64)" \
        -D TESTS=ON \
        %{nil}
%make_build %{_vpath_builddir}-8bytes_int
%endif

Just found that lapack last version has bug in lapack.pc.in has hardcoded -llapack and to fix that it is necessary to add

-Libs: -L${libdir} -llapack
+Libs: -L${libdir} -l@LAPACKLIB@

To have Libs: -L${libdir} -llapack in generated lapack.pc and Libs: -L${libdir} -llapack64 in generated lapack64.pc.

I found as well many LTO related issue in flexiblas but will open for that separated ticket. If everything will be oOK after fixing above shortly should have correctly build flexiblas and shortly will be able to close this ticket πŸ˜„

Thank you very much for all your hits πŸ‘ πŸ˜„

kloczek commented 3 months ago

FYI just created https://github.com/Reference-LAPACK/lapack/pull/1015

grisuthedragon commented 3 months ago

Since this seems to not a bug in FlexiBLAS, i will close this issue.

kloczek commented 3 months ago

OK just been able successfully build flexiblas πŸ˜„ Thank you very one more time much for all your help.

Last thing is that on test suite execution ech unit is executed with multiple threads which number is probably calculated on number of available cores. Result is that more cores has build system than more threads is executed. All ctest units are executed in parallel and it causes that on 24/48 cores box load in peak was +2k and it took ordrr of magnitude longer to finish it than running all unit sequentially. This can be tuned using https://www.scivision.dev/cmake-resource-lock-ctest/ Do you want for that separated ticket? πŸ€”

grisuthedragon commented 3 months ago

That's is exactly the behavior I expected when running ctest in parallel and having any multithreaded BLAS set as default. If you are using NETLIB Blas as default, no oversubscription will happen. It is the same problem as you run many compute intensive applications on your computer. Thus no bug.

kloczek commented 3 months ago

That's is exactly the behavior I expected when running ctest in parallel

That is default setup of the ctest and as single unit executed without parallelization consumes 100% of ALL cores it means that this should be changes to sequential execution.

grisuthedragon commented 3 months ago

Since my default workflow uses NETLIB (i.e. a single threaded BLAS) in testing, I am used to running the test in parallel. For any multithreaded BLAS you have to ensure that the number of threads do not lead to an oversubscription. It is the same as running many computational expensive applications at once on a single computer.

In your case, a single test consumes all of the available CPU resources since you are using OpenBLAS with OpenMP. That will also happen if you are compiling LAPACK with the BLAS operations coming from OpenBLAS.

kloczek commented 3 months ago

Are you sure? πŸ€” As you may see my cmake options I';m not changing default build end tested configuration (changed one comes only to the packaged resources) and I see that every unit consumes 100% of user time on all CPU cores.

grisuthedragon commented 3 months ago

I tested it with cmake ( cmake-3.28.2) from FC40, FlexiBLAS with NETLIB and running ctest utilizes one CPU core (out of 6, overall time 168.94 sec). Running ctest -j 6 I get all 6 cores utilized with the tests (Overall time 39.68 sec) . Everyting without oversubscription or similar problems.

kloczek commented 3 months ago

Please try with those cmake setting which I'm unsung.

grisuthedragon commented 3 months ago

I did and the above thing happend.