Open boegel opened 3 years ago
Can you provide the backtrace with debug information? How does it look like in valgrind?
Backtrace with debug info:
#0 dnrm2_k (n=2, x=<optimized out>, inc_x=1) at ../kernel/power/../arm/nrm2.c:69
#1 0x00007ffff453d788 in dnrm2_ (N=<optimized out>, x=<optimized out>, INCX=<optimized out>) at nrm2.c:61
#2 0x00007ffff62cf9fc in dnrm2_ (n=<optimized out>, x=<optimized out>, incx=<optimized out>) at /tmp/centos/FlexiBLAS/3.0.4/GCC-10.3.0/flexiblas-3.0.4/src/wrapper_blas_gnu.c:2899
#3 0x00007ffff4d788ec in dgeev (jobvl=..., jobvr=..., n=2, a=..., lda=<optimized out>, wr=..., wi=..., vl=..., ldvl=2, vr=..., ldvr=2, work=..., lwork=260, info=<optimized out>, _jobvl=140737323525740, _jobvr=8) at dgeev.f:490
#4 0x00007ffff639e594 in dgeev_ (jobvl=0x7ffffffe655c "NV", jobvr=0x7ffffffe655d "V", n=0x7ffffffe6548, a=0x7fff650fc3a0, lda=0x7ffffffe654c, wr=0x7fff650fc3c0, wi=0x7fff650fc3d0, vl=0x7fff650fc3e0, ldvl=0x7ffffffe6550, vr=0x7fff650fc3e0, ldvr=0x7ffffffe6554,
work=0x7fff650458d0, lwork=0x7ffffffe6558, info=0x7ffffffe6560) at /tmp/centos/FlexiBLAS/3.0.4/GCC-10.3.0/flexiblas-3.0.4/src/lapack_interface/wrapper/dgeev.c:80
#5 0x00007fff7364b334 in call_dgeev (params=0x7ffffffe6500) at numpy/linalg/umath_linalg.c.src:2292
#6 DOUBLE_eig_wrapper (JOBVL=JOBVL@entry=78 'N', JOBVR=JOBVR@entry=86 'V', args=0x7fff5142d4a0, dimensions=<optimized out>, steps=<optimized out>) at numpy/linalg/umath_linalg.c.src:2292
#7 0x00007fff7364c02c in DOUBLE_eig (args=<optimized out>, dimensions=<optimized out>, steps=<optimized out>, __NPY_UNUSED_TAGGEDfunc=<optimized out>) at numpy/linalg/umath_linalg.c.src:2336
#8 0x00007ffff6a5d294 in PyUFunc_GeneralizedFunction (op=0x7ffffffe8270, kwds=0x0, args=0x7fff5142d470, ufunc=0x0) at numpy/core/src/umath/ufunc_object.c:2986
#9 PyUFunc_GenericFunction_int (ufunc=<optimized out>, ufunc@entry=0x7fff736c1130, args=args@entry=0x7fff5005aca0, kwds=kwds@entry=0x7fff50e7a700, op=op@entry=0x7ffffffe8270) at numpy/core/src/umath/ufunc_object.c:3119
#10 0x00007ffff6a5f740 in ufunc_generic_call (ufunc=0x7fff736c1130, args=0x7fff5005aca0, kwds=0x7fff50e7a700) at numpy/core/src/umath/ufunc_object.c:4747
...
I'll look into valgrind too.
@grisuthedragon No segmentation fault when running via Valgrind it seems (though a bunch of unrelated "Invalid read of size 4
" cases in Python itself are reported). So that's a dead end I think, I'm afraid...
That's weird. I try to compile FB + Numpy on my power system asap.
To quickly trigger the segfault, you can use python -c "import numpy as np; np.linalg.test()"
.
I tried this too on a real ppc machine and the minimal reproducer for "issues" I got is python -c "import numpy as np; np.linalg.test(verbose=3, extra_argv=['-k', 'TestEigvals and test_sq_cases'])"
which either segfaults with a double free or fails the test (works with OpenBLAS directly)
I also see messages in stderr:
** On entry to DGEHRD parameter number 8 had an illegal value
** On entry to DGEHRD parameter number 8 had an illegal value
** On entry to DORGHR parameter number 8 had an illegal value
** On entry to DGEHRD parameter number 8 had an illegal value
** On entry to DGEHRD parameter number 8 had an illegal value
** On entry to DORGHR parameter number 8 had an illegal value
** On entry to DGEHRD parameter number 8 had an illegal value
** On entry to DGEHRD parameter number 8 had an illegal value
** On entry to DORGHR parameter number 8 had an illegal value
** On entry to ZGEHRD parameter number 5 had an illegal value
** On entry to ZHSEQR parameter number 7 had an illegal value
Those are from the numpy xerblas error handler and I guess those are a good hint on to the real problem
More minimal reproducer: python -c "from numpy import array, linalg; linalg.eigvals(array([[1., 2.], [3., 4.]]))"
I suspect a stackoverflow due to GCC misoptimizing OpenBLAS which becomes apparent by FlexiBLAS as FlexiBLAS uses a the stack to save a register which gets overwritten by the bug. I reported this as https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100799
@Flamefire Thanks for the work and identifying, where this behaviour comes from. Lets wait until the gcc guys react and see how they see this problem.
The IBM compiler guys are looking into this. It seems to be indeed a compiler issue since GCC 7. So I'd say this can be closed as there is nothing short of providing a better error message that can be done here
@Flamefire Any updates on this?
Small update here from our side: we've side-stepped this problem by compiling OpenBLAS with -fstack-protector-strong
on POWER, see https://github.com/easybuilders/easybuild-easyconfigs/pull/15885 for more information
The GCC developers determined this a bug in the usage related to the Fortran calling convention:
As described in (https://gcc.gnu.org/onlinedocs/gfortran/Argument-passing-conventions.html), since the first parameter to DGEBAL is of type CHARACTER, there is an extra hidden argument. Change the call to DGEBAL from dgebal (the flexiBLAS wrapper routine) to take an extra argument. This causes the compiler to allocate a parameter save area in dgebal's frame, as there are now 9 parameters but only 8 parameter registers.
@Flamefire I know about this extra arguments, but due to compatibility reasons in the early times of FlexiBLAS, we neglected them. Even using CBLAS/LAPACKE from the reference implementation can lead to this issue, since they "forget" about these additional parameters as well.
For FlexiBLAS I will do some tests and, if successful, integrate it in the next release.
I'm seeing a
Segmentation fault
when running thenumpy
1.20.3 tests when using FlexiBLAS 3.0.4 with OpenBLAS 0.3.15, but not when linking to OpenBLAS 0.3.15 directly, which tells me FlexiBLAS is somehow causing the segmentation fault...I'm not seeing this problem on Intel (Haswell, Skylake X), AMD (Rome), or Arm (AWS Graviton2).
Here's a partial backtrace I obtained when running the numpy tests via
gdb
:This only happens when
numpy
is linked withFlexiBLAS
:Any ideas on what may be causing this segmentation fault?
I tried using
ulimit -s unlimited
(default is8192
on that system), no change.After
export FLEXIBLAS=netlib
to make FlexiBLAS use the fallbacknetlib
backend, the segmentation fault doesn't happen either...