libflame and R

[sgsokol]

I am trying to replace standard lapack library in R by libflame to see if any gain can be obtained. Unfortunately, libflames.so fails to load with the following error message:

/usr/local/R-3.4.3_gcc/lib64/R/lib/libRlapack.so: undefined symbol: blas_zhemv2_x_

Don't be confused, here, libRlapack.so is just a symlink to libflame.so which was compiled with

./configure EXTRA_CFLAGS="-O3 -mtune=native -march=native" LDFLAGS=-L/opt/OpenBLAS/lib LIBS=-lopenblas --enable-dynamic-build --disable-static-build --enable-lapack2flame --enable-supermatrix --enable-memory-alignment=8 --enable-max-arg-list-hack

The lacking symbol blas_zhemv2_x_ is used in calls found in map/lapack2flamec/f2c/c/zla_herfsx_extended.c and map/lapack2flamec/f2c/c/zla_porfsx_extended.c which seems to correspond to xblas. If xblas is not used, shouldn't these files be excluded from lapack mapping?

[fgvanzee]

@sgsokol Yes, I agree that unused symbols should not be referenced. This issue probably went unnoticed for so long because very few people use a shared library builds of libflame.

I'll look into it.

[fgvanzee]

@sgsokol Try out 5151689, which disables the code found in the map/lapack2flamec/f2c/c/?la_*_extended.c files. Hopefully that takes care of all of the unresolved symbols, but if there are more, we'll iterate until they're all gone. Thanks for your bug report.

[sgsokol]

Thanks for the patch and possibility to iterate. This time I have:

/usr/local/R-3.4.3_gcc/lib64/R/lib/libRlapack.so: undefined symbol: cla_syrfsx_extended_

[fgvanzee]

@sgsokol Yes, of course--I disabled the code that defines the xblas functions but forgot to disable the code that calls those functions. I'll work on it.

Basically, I have to prune the entire reverse-dependency tree. This will take some time.

[fgvanzee]

@sgsokol Okay, give 16b5591 a try. Thanks for your patience.

[sgsokol]

Next round ;)

/usr/local/R-3.4.3_gcc/lib64/R/lib/libRlapack.so: undefined symbol: do_lio

[sgsokol]

Oups... After some searching it's a problem of libf2c.so. It was not installed on my system (may be ./configure should test for it?). After installation, I've got a lacking MAIN__ symbol which I had to disable manually. Now, it compiles well with LIBS="-lopenblas -u MAIN__ -lf2c" and it loads well in R too. Now I have to do some tests to see if any acceleration can be detected.

[fgvanzee]

@sgsokol Our goal is to obviate libf2c. We do not want libflame to need it. My understanding was that this was feasible, so long as libflame provided all of the parts of libf2c that are needed, but I am not an expert on this particular topic, so I could be wrong.

BTW, I would test the linking myself, but I'm having trouble on my end. I get countless undefined reference errors about symbols that are present in libflame. I'm not sure what is causing it.

[fgvanzee]

@sgsokol I have found an implementation of libf2c [1]. I will try to import just the components needed in order to satisfy the symbols in libflame. (Hopefully this is not a a fool's errand.)

As for the MAIN__ symbol, it should not be there. I will fix that too.

[1] https://github.com/juanjosegarciaripoll/f2c/

[fgvanzee]

@sgsokol I managed to make progress with linking the libflame testsuite driver to libflame.so. We are down to just three missing symbols:

/u/field/flame/lib/libflame.so: undefined reference to `do_lio'
/u/field/flame/lib/libflame.so: undefined reference to `s_wsle'
/u/field/flame/lib/libflame.so: undefined reference to `e_wsle'
collect2: error: ld returned 1 exit status
Makefile:87: recipe for target 'test_libflame' failed
make: *** [test_libflame] Error 1

Note that I already removed the reference to the missing MAIN__ symbol. I will continue to work on this. Thanks again.

[fgvanzee]

@sgsokol I can now successfully link the test suite to libflame.so. Try 3bc0286 and let me know how it goes. (Note that you should no longer need -lf2c, nor do you need to accommodate the MAIN__ symbol. The only dependency that libflame should have now is libc/libm, which may be automatic in your case of linking with R, and -lpthreads if you enabled POSIX threads.)

[sgsokol]

It went well without -lf2c neither -u MAIN__. Now, in R, I have compared the timing of native lapack and libflame and found it very close, e.g. around 1.09 s for SVD of 1000x1000 matrix and around 0.19 s for QR of the same matrix. Do you have equal timings on your side too for those operations or have I missed some magic configuration parameter?

[fgvanzee]

@sgsokol First, sorry for the long delay. I composed a reply and forgot to send it! :( That reply is below.

Glad we finally got it working. Thanks for your help along the way.

I can't comment on your timings since we likely are using different hardware. If the underlying BLAS are optimized, there isn't much to tune in SVD, if I recall correctly.

Also, whatever numbers you get, keep in mind there is room for improvement in libflame with Hermitian EVD and general SVD because a key component (the application of Givens rotations) still uses older SSE instructions, whereas AVX/AVX2/AVX512 is now widely available. Furthermore, there are advanced "early aggressive deflation" techniques that can further accelerate these operations--research that has yet to be published. I hope to return to this someday, but for now I must tend to other priorities.