Addition to Gentoo Science and query on a BLAS providing library's structure

[epsilon-0]

I'm one of maintainer for the Gentoo Science repository and was hoping to add this package to the mix.
I saw that you had some instructions of how to use this on gentoo - https://www.mpi-magdeburg.mpg.de/2427308/flexiblas_install

That's pretty awesome!

I had a quick question about the structure of BLAS providers in general.
Is it mandated that the BLAS providing library (whichever it be) like ATLAS/OpenBLAS/CLBlast/BLIS/etc, needs to have all the symbols present in one SONAME library, like libopenblas or whatever?
Is it possible that some symbols are present in libcblas_provider.so and it loads another library libcblas_provider_helper.so that has the remaining symbols? Is the monolithic structure of all symbols being present in libblas.so.3 a mandated specification?
I understand that there might not be many(/any) libraries which do this, but I just wanted to know if this is a requirement by NetLib.

Thanks a lot, Aisha

Re adding to gentoo science - If you would rather be the maintainer for this in Gentoo, please let me know, or if you can give me the go ahead to add it would also be OK :smile_cat:
I seem to have seen some effort a few years ago but I don't think it went anywhere. We have been short on personnel in Gentoo and I just joined the team a few weeks back, so I'm trying to get the cool packages back in the repository :smiley_cat:

[grisuthedragon]

Hi @epsilon-0, great to read that FlexiBLAS gains more an more visibility. The Gentoo instructions are quite old and no longer maintained by my colleague when he was still an active gentoo user. But you can used it as base for your work. The included Debian/Ubuntu packaging could help how to realized this or the Fedora package could also give a bunch of hints (https://copr.fedorainfracloud.org/coprs/iucar/flexiblas/)

At the moment the BLAS provides must be fully integrated, i.e., a shared library containing all symbols. OpenBLAS in its default configuration fulfils this. If the provided library is not full-featured, it should at least contain the BLAS part. The missing objects are included in the following way:

• CBLAS is wrapped directly around the BLAS calls using a modified version of the reference implementation.
• Non Netlib-BLAS routines included in OpenBLAS or MKL are replaced with own Fortran codes, e.g. DAXPBY.
• LAPACK routines come from a fallback Netlib LAPACK implementation. (That coincides with the way OpenBLAS and ATLAS are doing. They only optimize a few routines and take the rest directly from NETLIB)

Is it possible that some symbols are present in libcblas_provider.so and it loads another library libcblas_provider_helper.so that has the remaining symbols?

Not yet, but it sound like an idea.

Is the monolithic structure of all symbols being present in libblas.so.3 a mandated specification?

No since missing functions are automatically replaced by fallback implementations.

I hope this helps. And I would be happy if we find a way into the official Gentoo repositories.

[Enchufa2]

@epsilon-0 If you happen to be familiar with SPEC files, this is how we currently do it for Fedora, and this is how one of our builds looks like. Hope it helps.

[epsilon-0]

Great! I'm happy to add this as well.

Is the monolithic structure of all symbols being present in libblas.so.3 a mandated specification?

No since missing functions are automatically replaced by fallback implementations.

Sorry I was unclear, I meant if this was mandated by official Netlibs specifications. I am aware that flexiblas does have fallback implementations so that isn't a problem with that, as flexiblas itself does have all symbols present in a single library, I wanted to know if the official documentation says that all need to be present.

Because I was playing around with ideas like symbol loading and runtime switching yesterday and wrote a tiny wrapper for a single function cblas_dgemm (https://bsd.ac/c2543cy) when I tried to load cblas_dgemm symbol from libmkl_rt.so, it does load the symbol but then complains about omp_get_num_procs symbol being missing. But the same wrapper works for OpenBLAS, BLIS, Netlib implementations.

So I was very confused about why that simple wrapper works with other but not Intel MKL.

[grisuthedragon]

There is no "official" Netlib Standard how BLAS and LAPACK have to look. For BLAS there are three papers from 1978 to 1990 defining what is level 1, 2, and 3. It is also not specified whether BLAS and LAPACK should be split into two libraries or not. If you ask the Debian Science Team, the will probably say: "Yes, they are two diffirent libraries". When you look at OpenBLAS it is one library by default ( the debian guys split it afterwards for example). What I learn from my work, a BLAS/LAPACK implementation should at least include all Fortran-symbols, that are created when you compile the reference implementation. For all additional routines, like domatcopy, ... the user have to check whether this is included in the BLAS library he want to use.

The dynamic loading is exactly what I does in FlexiBLAS so you do not need to reinvent it. But dealing with the MKL is more or less complicated. Especially when you use the mkl_rt version, which binary interface could be changed by the application at runtime. The way we interface the MKL very tricky. We have an intermediate dummy library which does nothing more than referencing all BLAS and LAPACK symbols, without containing any executable code and this one is linked against the MKL. When you now open this dummy library with dlopen, everything required for the MKL is loaded as well.

[epsilon-0]

THANK YOU!!!! I was sooo confused with all this and when I couldn't find documentation mandating this!! It is good to know that I was not being dumb.