search

kaldi-asr / kaldi

kaldi-asr/kaldi is the official location of the Kaldi project.
http://kaldi-asr.org
Other
14.33k stars 5.33k forks source link

Using system OpenBLAS libraries in Ubuntu #2946

Open danpovey opened 5 years ago

danpovey commented 5 years ago

Lately we have had various people reporting runtime errors like clapackdgetrf not defined, when using ubuntu 18.04. This is a simple configure-script problem where it fails to detect that a library called liblapack.so.3 is not really an ATLAS library, and I'm going to fix this separately.

But the bigger point is: maybe we should be trying to use the system-level OpenBLAS libraries rather than just asking for ATLAS, as people have reported that OpenBLAS is faster.
However, this doesn't interact right with Kaldi's build system. Kaldi currently expects there to be an OPENBLASROOT that contains lib/ and include/ dirs, where the lib/ dir contains libopenblas.a and the include/ dir contains cblas.h and lapacke.h, but the system packages on ubuntu 18.04 aren't structured that way. See below for what the package contains. Someone needs to sort this out and figure out what we need to include:

 apt-file list libopenblas-dev
<snips below>
libopenblas-dev: /usr/include/x86_64-linux-gnu/cblas-openblas.h
libopenblas-dev: /usr/include/x86_64-linux-gnu/f77blas.h
libopenblas-dev: /usr/include/x86_64-linux-gnu/openblas_config.h
libopenblas-dev: /usr/lib/x86_64-linux-gnu/libopenblas.so
libopenblas-dev: /usr/lib/x86_64-linux-gnu/openblas/libblas.so
libopenblas-dev: /usr/lib/x86_64-linux-gnu/openblas/liblapack.so

I don't know if one of these is the top-level one that we should actually include (e.g. perhaps /usr/lib/x8664-linux-gnu/libopenblas.so). I do notice symbols like sgetrf in libopenblas.so, which means it does provide those lapack symbols, even it doesn't declare them in the headers that it ships. The tools/CLAPACK directory may be relevant here: we might be able to use the standard CLAPACK headers that we distribute there. But I really have no idea. In any case, changes to Kaldi's configure scripts and maybe even its code may be required. The easiest thing might be to find a kaldi.mk that works and then later work on the configure script.

@galv do you have any time to work on this?

galv commented 5 years ago

Hi Dan,

I can probably work on using the OpenBLAS libraries, since I know the exact problem you are talking about. In fact, Centos7 has the same issue, where openblas is not installed to a single directory:

daniel@server5 ~ $ rpm -ql openblas-devel
/usr/include/openblas
/usr/include/openblas/cblas.h
/usr/include/openblas/f77blas.h
/usr/include/openblas/lapacke.h
/usr/include/openblas/lapacke_config.h
/usr/include/openblas/lapacke_mangling.h
/usr/include/openblas/lapacke_utils.h
/usr/include/openblas/openblas_config.h
/usr/lib64/libopenblas.so
/usr/lib64/libopenblas64.so
/usr/lib64/libopenblas64_.so
/usr/lib64/libopenblaso.so
/usr/lib64/libopenblaso64.so
/usr/lib64/libopenblaso64_.so
/usr/lib64/libopenblasp.so
/usr/lib64/libopenblasp64.so
/usr/lib64/libopenblasp64_.so

I noticed that problem when I worked on updating the tools/CLAPACK directory a few months ago, since atlas has the same problem (this on Centos7, but I am sure it is true of Debian machines as well):

daniel@server5 ~ $ rpm -ql atlas-devel
/usr/include/atlas
/usr/include/atlas-x86_64-base
/usr/include/atlas-x86_64-base/atlas_buildinfo.h
/usr/include/atlas-x86_64-base/atlas_cGetNB_gelqf.h
/usr/include/atlas-x86_64-base/atlas_cGetNB_geqlf.h
/usr/include/atlas-x86_64-base/atlas_cGetNB_geqrf.h
/usr/include/atlas-x86_64-base/atlas_cGetNB_gerqf.h
/usr/include/atlas-x86_64-base/atlas_cNCmm.h
/usr/include/atlas-x86_64-base/atlas_cacheedge.h
/usr/include/atlas-x86_64-base/atlas_cher2.h
/usr/include/atlas-x86_64-base/atlas_cher2_L1.h
/usr/include/atlas-x86_64-base/atlas_cher2_L2.h
/usr/include/atlas-x86_64-base/atlas_cmvn.h
/usr/include/atlas-x86_64-base/atlas_cmvn_L1.h
/usr/include/atlas-x86_64-base/atlas_cmvn_L2.h
/usr/include/atlas-x86_64-base/atlas_cmvnkernels.h
/usr/include/atlas-x86_64-base/atlas_cmvt.h
/usr/include/atlas-x86_64-base/atlas_cmvt_L1.h
/usr/include/atlas-x86_64-base/atlas_cmvt_L2.h
/usr/include/atlas-x86_64-base/atlas_cmvtkernels.h
/usr/include/atlas-x86_64-base/atlas_cr1.h
/usr/include/atlas-x86_64-base/atlas_cr1_L1.h
/usr/include/atlas-x86_64-base/atlas_cr1_L2.h
/usr/include/atlas-x86_64-base/atlas_cr1kernels.h
/usr/include/atlas-x86_64-base/atlas_cr2.h
/usr/include/atlas-x86_64-base/atlas_cr2_L1.h
/usr/include/atlas-x86_64-base/atlas_cr2_L2.h
/usr/include/atlas-x86_64-base/atlas_cr2kernels.h
/usr/include/atlas-x86_64-base/atlas_csNKB.h
/usr/include/atlas-x86_64-base/atlas_csyr.h
/usr/include/atlas-x86_64-base/atlas_csyr2NX.h
/usr/include/atlas-x86_64-base/atlas_csyrNX.h
/usr/include/atlas-x86_64-base/atlas_csyr_L1.h
/usr/include/atlas-x86_64-base/atlas_csyr_L2.h
/usr/include/atlas-x86_64-base/atlas_csysinfo.h
/usr/include/atlas-x86_64-base/atlas_ctGetNB_gelqf.h
/usr/include/atlas-x86_64-base/atlas_ctGetNB_geqlf.h
/usr/include/atlas-x86_64-base/atlas_ctGetNB_geqrf.h
/usr/include/atlas-x86_64-base/atlas_ctGetNB_gerqf.h
/usr/include/atlas-x86_64-base/atlas_ctrsmXover.h
/usr/include/atlas-x86_64-base/atlas_dGetNB_gelqf.h
/usr/include/atlas-x86_64-base/atlas_dGetNB_geqlf.h
/usr/include/atlas-x86_64-base/atlas_dGetNB_geqrf.h
/usr/include/atlas-x86_64-base/atlas_dGetNB_gerqf.h
/usr/include/atlas-x86_64-base/atlas_dNCmm.h
/usr/include/atlas-x86_64-base/atlas_dlamch.h
/usr/include/atlas-x86_64-base/atlas_dmvn.h
/usr/include/atlas-x86_64-base/atlas_dmvn_L1.h
/usr/include/atlas-x86_64-base/atlas_dmvn_L2.h
/usr/include/atlas-x86_64-base/atlas_dmvnkernels.h
/usr/include/atlas-x86_64-base/atlas_dmvt.h
/usr/include/atlas-x86_64-base/atlas_dmvt_L1.h
/usr/include/atlas-x86_64-base/atlas_dmvt_L2.h
/usr/include/atlas-x86_64-base/atlas_dmvtkernels.h
/usr/include/atlas-x86_64-base/atlas_dr1.h
/usr/include/atlas-x86_64-base/atlas_dr1_L1.h
/usr/include/atlas-x86_64-base/atlas_dr1_L2.h
/usr/include/atlas-x86_64-base/atlas_dr1kernels.h
/usr/include/atlas-x86_64-base/atlas_dr2.h
/usr/include/atlas-x86_64-base/atlas_dr2_L1.h
/usr/include/atlas-x86_64-base/atlas_dr2_L2.h
/usr/include/atlas-x86_64-base/atlas_dr2kernels.h
/usr/include/atlas-x86_64-base/atlas_dsyr.h
/usr/include/atlas-x86_64-base/atlas_dsyr2.h
/usr/include/atlas-x86_64-base/atlas_dsyr2NX.h
/usr/include/atlas-x86_64-base/atlas_dsyr2_L1.h
/usr/include/atlas-x86_64-base/atlas_dsyr2_L2.h
/usr/include/atlas-x86_64-base/atlas_dsyrNX.h
/usr/include/atlas-x86_64-base/atlas_dsyr_L1.h
/usr/include/atlas-x86_64-base/atlas_dsyr_L2.h
/usr/include/atlas-x86_64-base/atlas_dsysinfo.h
/usr/include/atlas-x86_64-base/atlas_dtGetNB_gelqf.h
/usr/include/atlas-x86_64-base/atlas_dtGetNB_geqlf.h
/usr/include/atlas-x86_64-base/atlas_dtGetNB_geqrf.h
/usr/include/atlas-x86_64-base/atlas_dtGetNB_gerqf.h
/usr/include/atlas-x86_64-base/atlas_dtrsmXover.h
/usr/include/atlas-x86_64-base/atlas_pthreads.h
/usr/include/atlas-x86_64-base/atlas_sGetNB_gelqf.h
/usr/include/atlas-x86_64-base/atlas_sGetNB_geqlf.h
/usr/include/atlas-x86_64-base/atlas_sGetNB_geqrf.h
/usr/include/atlas-x86_64-base/atlas_sGetNB_gerqf.h
/usr/include/atlas-x86_64-base/atlas_sNCmm.h
/usr/include/atlas-x86_64-base/atlas_slamch.h
/usr/include/atlas-x86_64-base/atlas_smvn.h
/usr/include/atlas-x86_64-base/atlas_smvn_L1.h
/usr/include/atlas-x86_64-base/atlas_smvn_L2.h
/usr/include/atlas-x86_64-base/atlas_smvnkernels.h
/usr/include/atlas-x86_64-base/atlas_smvt.h
/usr/include/atlas-x86_64-base/atlas_smvt_L1.h
/usr/include/atlas-x86_64-base/atlas_smvt_L2.h
/usr/include/atlas-x86_64-base/atlas_smvtkernels.h
/usr/include/atlas-x86_64-base/atlas_sr1.h
/usr/include/atlas-x86_64-base/atlas_sr1_L1.h
/usr/include/atlas-x86_64-base/atlas_sr1_L2.h
/usr/include/atlas-x86_64-base/atlas_sr1kernels.h
/usr/include/atlas-x86_64-base/atlas_sr2.h
/usr/include/atlas-x86_64-base/atlas_sr2_L1.h
/usr/include/atlas-x86_64-base/atlas_sr2_L2.h
/usr/include/atlas-x86_64-base/atlas_sr2kernels.h
/usr/include/atlas-x86_64-base/atlas_ssyr.h
/usr/include/atlas-x86_64-base/atlas_ssyr2.h
/usr/include/atlas-x86_64-base/atlas_ssyr2NX.h
/usr/include/atlas-x86_64-base/atlas_ssyr2_L1.h
/usr/include/atlas-x86_64-base/atlas_ssyr2_L2.h
/usr/include/atlas-x86_64-base/atlas_ssyrNX.h
/usr/include/atlas-x86_64-base/atlas_ssyr_L1.h
/usr/include/atlas-x86_64-base/atlas_ssyr_L2.h
/usr/include/atlas-x86_64-base/atlas_ssysinfo.h
/usr/include/atlas-x86_64-base/atlas_stGetNB_gelqf.h
/usr/include/atlas-x86_64-base/atlas_stGetNB_geqlf.h
/usr/include/atlas-x86_64-base/atlas_stGetNB_geqrf.h
/usr/include/atlas-x86_64-base/atlas_stGetNB_gerqf.h
/usr/include/atlas-x86_64-base/atlas_strsmXover.h
/usr/include/atlas-x86_64-base/atlas_taffinity.h
/usr/include/atlas-x86_64-base/atlas_tcacheedge.h
/usr/include/atlas-x86_64-base/atlas_trsmNB.h
/usr/include/atlas-x86_64-base/atlas_tsumm.h
/usr/include/atlas-x86_64-base/atlas_type.h
/usr/include/atlas-x86_64-base/atlas_zGetNB_gelqf.h
/usr/include/atlas-x86_64-base/atlas_zGetNB_geqlf.h
/usr/include/atlas-x86_64-base/atlas_zGetNB_geqrf.h
/usr/include/atlas-x86_64-base/atlas_zGetNB_gerqf.h
/usr/include/atlas-x86_64-base/atlas_zNCmm.h
/usr/include/atlas-x86_64-base/atlas_zdNKB.h
/usr/include/atlas-x86_64-base/atlas_zher2.h
/usr/include/atlas-x86_64-base/atlas_zher2_L1.h
/usr/include/atlas-x86_64-base/atlas_zher2_L2.h
/usr/include/atlas-x86_64-base/atlas_zmvn.h
/usr/include/atlas-x86_64-base/atlas_zmvn_L1.h
/usr/include/atlas-x86_64-base/atlas_zmvn_L2.h
/usr/include/atlas-x86_64-base/atlas_zmvnkernels.h
/usr/include/atlas-x86_64-base/atlas_zmvt.h
/usr/include/atlas-x86_64-base/atlas_zmvt_L1.h
/usr/include/atlas-x86_64-base/atlas_zmvt_L2.h
/usr/include/atlas-x86_64-base/atlas_zmvtkernels.h
/usr/include/atlas-x86_64-base/atlas_zr1.h
/usr/include/atlas-x86_64-base/atlas_zr1_L1.h
/usr/include/atlas-x86_64-base/atlas_zr1_L2.h
/usr/include/atlas-x86_64-base/atlas_zr1kernels.h
/usr/include/atlas-x86_64-base/atlas_zr2.h
/usr/include/atlas-x86_64-base/atlas_zr2_L1.h
/usr/include/atlas-x86_64-base/atlas_zr2_L2.h
/usr/include/atlas-x86_64-base/atlas_zr2kernels.h
/usr/include/atlas-x86_64-base/atlas_zsyr.h
/usr/include/atlas-x86_64-base/atlas_zsyr2NX.h
/usr/include/atlas-x86_64-base/atlas_zsyrNX.h
/usr/include/atlas-x86_64-base/atlas_zsyr_L1.h
/usr/include/atlas-x86_64-base/atlas_zsyr_L2.h
/usr/include/atlas-x86_64-base/atlas_zsysinfo.h
/usr/include/atlas-x86_64-base/atlas_ztGetNB_gelqf.h
/usr/include/atlas-x86_64-base/atlas_ztGetNB_geqlf.h
/usr/include/atlas-x86_64-base/atlas_ztGetNB_geqrf.h
/usr/include/atlas-x86_64-base/atlas_ztGetNB_gerqf.h
/usr/include/atlas-x86_64-base/atlas_ztrsmXover.h
/usr/include/atlas-x86_64-base/cXover.h
/usr/include/atlas-x86_64-base/cmm.h
/usr/include/atlas-x86_64-base/dXover.h
/usr/include/atlas-x86_64-base/dmm.h
/usr/include/atlas-x86_64-base/sXover.h
/usr/include/atlas-x86_64-base/smm.h
/usr/include/atlas-x86_64-base/zXover.h
/usr/include/atlas-x86_64-base/zmm.h
/usr/include/cblas.h
/usr/include/clapack.h
/usr/lib64/atlas/libsatlas.so
/usr/lib64/atlas/libtatlas.so
/usr/share/doc/atlas-devel-3.10.1
/usr/share/doc/atlas-devel-3.10.1/doc
/usr/share/doc/atlas-devel-3.10.1/doc/AtlasCredits.txt
/usr/share/doc/atlas-devel-3.10.1/doc/BootSequence.txt
/usr/share/doc/atlas-devel-3.10.1/doc/ChangeLog
/usr/share/doc/atlas-devel-3.10.1/doc/DirStruct.txt
/usr/share/doc/atlas-devel-3.10.1/doc/INDEX.txt
/usr/share/doc/atlas-devel-3.10.1/doc/LibReadme.txt
/usr/share/doc/atlas-devel-3.10.1/doc/README.dist
/usr/share/doc/atlas-devel-3.10.1/doc/TestTime.txt
/usr/share/doc/atlas-devel-3.10.1/doc/TroubleShoot.txt
/usr/share/doc/atlas-devel-3.10.1/doc/Windows.txt
/usr/share/doc/atlas-devel-3.10.1/doc/atlas_contrib.pdf
/usr/share/doc/atlas-devel-3.10.1/doc/atlas_devel.pdf
/usr/share/doc/atlas-devel-3.10.1/doc/atlas_install.pdf
/usr/share/doc/atlas-devel-3.10.1/doc/atlas_over.pdf
/usr/share/doc/atlas-devel-3.10.1/doc/cblas.pdf
/usr/share/doc/atlas-devel-3.10.1/doc/cblasqref.pdf
/usr/share/doc/atlas-devel-3.10.1/doc/f77blasqref.pdf
/usr/share/doc/atlas-devel-3.10.1/doc/lapackqref.pdf

Using the standard CLPACK headers we put in tools/CLAPACK probably works because I don't see why BLAS and LAPACK implementations would not be ABI compatible. To be honest, though, if I work on this, I'd like to use it as an opportunity to kill that directory, and instead depend on the headers provided by the "devel" installations of openblas and atlas.

But yes, I can work on this. I'm on vacation until Monday evening or so.

danpovey commented 5 years ago

It would be good if you could work on it.

But if there isn't a good reason to use the "system" headers I'd rather use the standard ones in the CLAPACK directory. The reason is: if we use the system headers it constrains how the package can be installed (we need to use the "dev" version), and we also have to worry about a lot more things in that case, i.e. we need to locate where the headers were installed and verify that they came from the corresponding package (which is nontrivial because different packages may sometimes use the same name for a header). There may be variation in these system headers that would cause us problems. Since the libraries are designed to be linked against by programs that may not even be checking the headers, it may be reasonable to rely on the expected binary compatibility.

Dan

On Fri, Dec 28, 2018 at 4:31 PM Daniel Galvez notifications@github.com wrote:

Hi Dan,

I can probably work on using the OpenBLAS libraries, since I know the exact problem you are talking about. In fact, Centos7 has the same issue, where openblas is not installed to a single directory:

daniel@server5 ~ $ rpm -ql openblas-devel /usr/include/openblas /usr/include/openblas/cblas.h /usr/include/openblas/f77blas.h /usr/include/openblas/lapacke.h /usr/include/openblas/lapacke_config.h /usr/include/openblas/lapacke_mangling.h /usr/include/openblas/lapacke_utils.h /usr/include/openblas/openblasconfig.h /usr/lib64/libopenblas.so /usr/lib64/libopenblas64.so /usr/lib64/libopenblas64.so /usr/lib64/libopenblaso.so /usr/lib64/libopenblaso64.so /usr/lib64/libopenblaso64.so /usr/lib64/libopenblasp.so /usr/lib64/libopenblasp64.so /usr/lib64/libopenblasp64.so

I noticed that problem when I worked on updating the tools/CLAPACK directory a few months ago, since atlas has the same problem (this on Centos7, but I am sure it is true of Debian machines as well):

daniel@server5 ~ $ rpm -ql atlas-devel /usr/include/atlas /usr/include/atlas-x86_64-base /usr/include/atlas-x86_64-base/atlas_buildinfo.h /usr/include/atlas-x86_64-base/atlas_cGetNB_gelqf.h /usr/include/atlas-x86_64-base/atlas_cGetNB_geqlf.h /usr/include/atlas-x86_64-base/atlas_cGetNB_geqrf.h /usr/include/atlas-x86_64-base/atlas_cGetNB_gerqf.h /usr/include/atlas-x86_64-base/atlas_cNCmm.h /usr/include/atlas-x86_64-base/atlas_cacheedge.h /usr/include/atlas-x86_64-base/atlas_cher2.h /usr/include/atlas-x86_64-base/atlas_cher2_L1.h /usr/include/atlas-x86_64-base/atlas_cher2_L2.h /usr/include/atlas-x86_64-base/atlas_cmvn.h /usr/include/atlas-x86_64-base/atlas_cmvn_L1.h /usr/include/atlas-x86_64-base/atlas_cmvn_L2.h /usr/include/atlas-x86_64-base/atlas_cmvnkernels.h /usr/include/atlas-x86_64-base/atlas_cmvt.h /usr/include/atlas-x86_64-base/atlas_cmvt_L1.h /usr/include/atlas-x86_64-base/atlas_cmvt_L2.h /usr/include/atlas-x86_64-base/atlas_cmvtkernels.h /usr/include/atlas-x86_64-base/atlas_cr1.h /usr/include/atlas-x86_64-base/atlas_cr1_L1.h /usr/include/atlas-x86_64-base/atlas_cr1_L2.h /usr/include/atlas-x86_64-base/atlas_cr1kernels.h /usr/include/atlas-x86_64-base/atlas_cr2.h /usr/include/atlas-x86_64-base/atlas_cr2_L1.h /usr/include/atlas-x86_64-base/atlas_cr2_L2.h /usr/include/atlas-x86_64-base/atlas_cr2kernels.h /usr/include/atlas-x86_64-base/atlas_csNKB.h /usr/include/atlas-x86_64-base/atlas_csyr.h /usr/include/atlas-x86_64-base/atlas_csyr2NX.h /usr/include/atlas-x86_64-base/atlas_csyrNX.h /usr/include/atlas-x86_64-base/atlas_csyr_L1.h /usr/include/atlas-x86_64-base/atlas_csyr_L2.h /usr/include/atlas-x86_64-base/atlas_csysinfo.h /usr/include/atlas-x86_64-base/atlas_ctGetNB_gelqf.h /usr/include/atlas-x86_64-base/atlas_ctGetNB_geqlf.h /usr/include/atlas-x86_64-base/atlas_ctGetNB_geqrf.h /usr/include/atlas-x86_64-base/atlas_ctGetNB_gerqf.h /usr/include/atlas-x86_64-base/atlas_ctrsmXover.h /usr/include/atlas-x86_64-base/atlas_dGetNB_gelqf.h /usr/include/atlas-x86_64-base/atlas_dGetNB_geqlf.h /usr/include/atlas-x86_64-base/atlas_dGetNB_geqrf.h /usr/include/atlas-x86_64-base/atlas_dGetNB_gerqf.h /usr/include/atlas-x86_64-base/atlas_dNCmm.h /usr/include/atlas-x86_64-base/atlas_dlamch.h /usr/include/atlas-x86_64-base/atlas_dmvn.h /usr/include/atlas-x86_64-base/atlas_dmvn_L1.h /usr/include/atlas-x86_64-base/atlas_dmvn_L2.h /usr/include/atlas-x86_64-base/atlas_dmvnkernels.h /usr/include/atlas-x86_64-base/atlas_dmvt.h /usr/include/atlas-x86_64-base/atlas_dmvt_L1.h /usr/include/atlas-x86_64-base/atlas_dmvt_L2.h /usr/include/atlas-x86_64-base/atlas_dmvtkernels.h /usr/include/atlas-x86_64-base/atlas_dr1.h /usr/include/atlas-x86_64-base/atlas_dr1_L1.h /usr/include/atlas-x86_64-base/atlas_dr1_L2.h /usr/include/atlas-x86_64-base/atlas_dr1kernels.h /usr/include/atlas-x86_64-base/atlas_dr2.h /usr/include/atlas-x86_64-base/atlas_dr2_L1.h /usr/include/atlas-x86_64-base/atlas_dr2_L2.h /usr/include/atlas-x86_64-base/atlas_dr2kernels.h /usr/include/atlas-x86_64-base/atlas_dsyr.h /usr/include/atlas-x86_64-base/atlas_dsyr2.h /usr/include/atlas-x86_64-base/atlas_dsyr2NX.h /usr/include/atlas-x86_64-base/atlas_dsyr2_L1.h /usr/include/atlas-x86_64-base/atlas_dsyr2_L2.h /usr/include/atlas-x86_64-base/atlas_dsyrNX.h /usr/include/atlas-x86_64-base/atlas_dsyr_L1.h /usr/include/atlas-x86_64-base/atlas_dsyr_L2.h /usr/include/atlas-x86_64-base/atlas_dsysinfo.h /usr/include/atlas-x86_64-base/atlas_dtGetNB_gelqf.h /usr/include/atlas-x86_64-base/atlas_dtGetNB_geqlf.h /usr/include/atlas-x86_64-base/atlas_dtGetNB_geqrf.h /usr/include/atlas-x86_64-base/atlas_dtGetNB_gerqf.h /usr/include/atlas-x86_64-base/atlas_dtrsmXover.h /usr/include/atlas-x86_64-base/atlas_pthreads.h /usr/include/atlas-x86_64-base/atlas_sGetNB_gelqf.h /usr/include/atlas-x86_64-base/atlas_sGetNB_geqlf.h /usr/include/atlas-x86_64-base/atlas_sGetNB_geqrf.h /usr/include/atlas-x86_64-base/atlas_sGetNB_gerqf.h /usr/include/atlas-x86_64-base/atlas_sNCmm.h /usr/include/atlas-x86_64-base/atlas_slamch.h /usr/include/atlas-x86_64-base/atlas_smvn.h /usr/include/atlas-x86_64-base/atlas_smvn_L1.h /usr/include/atlas-x86_64-base/atlas_smvn_L2.h /usr/include/atlas-x86_64-base/atlas_smvnkernels.h /usr/include/atlas-x86_64-base/atlas_smvt.h /usr/include/atlas-x86_64-base/atlas_smvt_L1.h /usr/include/atlas-x86_64-base/atlas_smvt_L2.h /usr/include/atlas-x86_64-base/atlas_smvtkernels.h /usr/include/atlas-x86_64-base/atlas_sr1.h /usr/include/atlas-x86_64-base/atlas_sr1_L1.h /usr/include/atlas-x86_64-base/atlas_sr1_L2.h /usr/include/atlas-x86_64-base/atlas_sr1kernels.h /usr/include/atlas-x86_64-base/atlas_sr2.h /usr/include/atlas-x86_64-base/atlas_sr2_L1.h /usr/include/atlas-x86_64-base/atlas_sr2_L2.h /usr/include/atlas-x86_64-base/atlas_sr2kernels.h /usr/include/atlas-x86_64-base/atlas_ssyr.h /usr/include/atlas-x86_64-base/atlas_ssyr2.h /usr/include/atlas-x86_64-base/atlas_ssyr2NX.h /usr/include/atlas-x86_64-base/atlas_ssyr2_L1.h /usr/include/atlas-x86_64-base/atlas_ssyr2_L2.h /usr/include/atlas-x86_64-base/atlas_ssyrNX.h /usr/include/atlas-x86_64-base/atlas_ssyr_L1.h /usr/include/atlas-x86_64-base/atlas_ssyr_L2.h /usr/include/atlas-x86_64-base/atlas_ssysinfo.h /usr/include/atlas-x86_64-base/atlas_stGetNB_gelqf.h /usr/include/atlas-x86_64-base/atlas_stGetNB_geqlf.h /usr/include/atlas-x86_64-base/atlas_stGetNB_geqrf.h /usr/include/atlas-x86_64-base/atlas_stGetNB_gerqf.h /usr/include/atlas-x86_64-base/atlas_strsmXover.h /usr/include/atlas-x86_64-base/atlas_taffinity.h /usr/include/atlas-x86_64-base/atlas_tcacheedge.h /usr/include/atlas-x86_64-base/atlas_trsmNB.h /usr/include/atlas-x86_64-base/atlas_tsumm.h /usr/include/atlas-x86_64-base/atlas_type.h /usr/include/atlas-x86_64-base/atlas_zGetNB_gelqf.h /usr/include/atlas-x86_64-base/atlas_zGetNB_geqlf.h /usr/include/atlas-x86_64-base/atlas_zGetNB_geqrf.h /usr/include/atlas-x86_64-base/atlas_zGetNB_gerqf.h /usr/include/atlas-x86_64-base/atlas_zNCmm.h /usr/include/atlas-x86_64-base/atlas_zdNKB.h /usr/include/atlas-x86_64-base/atlas_zher2.h /usr/include/atlas-x86_64-base/atlas_zher2_L1.h /usr/include/atlas-x86_64-base/atlas_zher2_L2.h /usr/include/atlas-x86_64-base/atlas_zmvn.h /usr/include/atlas-x86_64-base/atlas_zmvn_L1.h /usr/include/atlas-x86_64-base/atlas_zmvn_L2.h /usr/include/atlas-x86_64-base/atlas_zmvnkernels.h /usr/include/atlas-x86_64-base/atlas_zmvt.h /usr/include/atlas-x86_64-base/atlas_zmvt_L1.h /usr/include/atlas-x86_64-base/atlas_zmvt_L2.h /usr/include/atlas-x86_64-base/atlas_zmvtkernels.h /usr/include/atlas-x86_64-base/atlas_zr1.h /usr/include/atlas-x86_64-base/atlas_zr1_L1.h /usr/include/atlas-x86_64-base/atlas_zr1_L2.h /usr/include/atlas-x86_64-base/atlas_zr1kernels.h /usr/include/atlas-x86_64-base/atlas_zr2.h /usr/include/atlas-x86_64-base/atlas_zr2_L1.h /usr/include/atlas-x86_64-base/atlas_zr2_L2.h /usr/include/atlas-x86_64-base/atlas_zr2kernels.h /usr/include/atlas-x86_64-base/atlas_zsyr.h /usr/include/atlas-x86_64-base/atlas_zsyr2NX.h /usr/include/atlas-x86_64-base/atlas_zsyrNX.h /usr/include/atlas-x86_64-base/atlas_zsyr_L1.h /usr/include/atlas-x86_64-base/atlas_zsyr_L2.h /usr/include/atlas-x86_64-base/atlas_zsysinfo.h /usr/include/atlas-x86_64-base/atlas_ztGetNB_gelqf.h /usr/include/atlas-x86_64-base/atlas_ztGetNB_geqlf.h /usr/include/atlas-x86_64-base/atlas_ztGetNB_geqrf.h /usr/include/atlas-x86_64-base/atlas_ztGetNB_gerqf.h /usr/include/atlas-x86_64-base/atlas_ztrsmXover.h /usr/include/atlas-x86_64-base/cXover.h /usr/include/atlas-x86_64-base/cmm.h /usr/include/atlas-x86_64-base/dXover.h /usr/include/atlas-x86_64-base/dmm.h /usr/include/atlas-x86_64-base/sXover.h /usr/include/atlas-x86_64-base/smm.h /usr/include/atlas-x86_64-base/zXover.h /usr/include/atlas-x86_64-base/zmm.h /usr/include/cblas.h /usr/include/clapack.h /usr/lib64/atlas/libsatlas.so /usr/lib64/atlas/libtatlas.so /usr/share/doc/atlas-devel-3.10.1 /usr/share/doc/atlas-devel-3.10.1/doc /usr/share/doc/atlas-devel-3.10.1/doc/AtlasCredits.txt /usr/share/doc/atlas-devel-3.10.1/doc/BootSequence.txt /usr/share/doc/atlas-devel-3.10.1/doc/ChangeLog /usr/share/doc/atlas-devel-3.10.1/doc/DirStruct.txt /usr/share/doc/atlas-devel-3.10.1/doc/INDEX.txt /usr/share/doc/atlas-devel-3.10.1/doc/LibReadme.txt /usr/share/doc/atlas-devel-3.10.1/doc/README.dist /usr/share/doc/atlas-devel-3.10.1/doc/TestTime.txt /usr/share/doc/atlas-devel-3.10.1/doc/TroubleShoot.txt /usr/share/doc/atlas-devel-3.10.1/doc/Windows.txt /usr/share/doc/atlas-devel-3.10.1/doc/atlas_contrib.pdf /usr/share/doc/atlas-devel-3.10.1/doc/atlas_devel.pdf /usr/share/doc/atlas-devel-3.10.1/doc/atlas_install.pdf /usr/share/doc/atlas-devel-3.10.1/doc/atlas_over.pdf /usr/share/doc/atlas-devel-3.10.1/doc/cblas.pdf /usr/share/doc/atlas-devel-3.10.1/doc/cblasqref.pdf /usr/share/doc/atlas-devel-3.10.1/doc/f77blasqref.pdf /usr/share/doc/atlas-devel-3.10.1/doc/lapackqref.pdf

Using the standard CLPACK headers we put in tools/CLAPACK probably works because I don't see why BLAS and LAPACK implementations would not be ABI compatible. To be honest, though, if I work on this, I'd like to use it as an opportunity to kill that directory, and instead depend on the headers provided by the "devel" installations of openblas and atlas.

But yes, I can work on this. I'm on vacation until Monday evening or so.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/2946#issuecomment-450447552, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVu5y_YW7KH7cxsjWZ3feomDZ6AiBVks5u9rfwgaJpZM4ZkUKV .

kkm000 commented 5 years ago

X-ref: #3078.

kkm000 commented 5 years ago

@galv, we seem to be moving into making MKL the default BLAS library. Maybe it is not even worth it to spend time on this at the moment? MKL works out of the box, and it's actually a super-performer. On Mac (and Windows) it requires filling a form on Intel's site to download an installer, and for Linux it's a repo setup for packages, that I've already scripted. Check #3078 and decide. In any case. I think the best is to see if the push towards MKL will be non-problematic in the end, and then decide whether this issue needs more effort on our side.

ATLAS and OpenBlas packages are borked on debian too, you guessed it absolutely right! atlas-devel comes with the clapack.h file that has #include <cblas.h> in it, but the second file is renamed cblas-atlas.h. so the includes do not even work out of the box. OpenBLAS and cblas-devel packages also have their cblas.h file renamed with a package name prefix. It's a mess. I remember Ubuntu 14.04 packages all placed cblas.h into /usr/include, but all included each other in the "conflicts with" field, so only one could have been installed. At some moment, they changed that by renaming cblas.h files.

The only sensible solution I can think of is create a single-line include file named cblas.h (which in turn would include a correctly named file) in a place where there are no other include files (e.g. src/), and tuck that path to the end of compile's include path, so it's looked up for the <> includes the latest (needed to salvage ATLAS debian packaging; we could use the "" includes in our code but that would not help). But again, this may be put now on backburner, in my opinion.

danpovey commented 5 years ago

In my opinion it was a mistake to attempt to use the system-provided includes, because the includes that we ship with Kaldi (or download) work just fine. I mean, it was solving a problem that didn't exist, just for the sake of being theoretically the right way to do things.

But I do think it's a good idea to retain the ability to use OpenBLAS; there might be systems where it's faster than MLK, and I don't want our code to become dependent on peculiarities of how MLK works.

On Sun, Mar 31, 2019 at 6:21 AM kkm (aka Kirill Katsnelson) < notifications@github.com> wrote:

@galv https://github.com/galv, we seem to be moving into making MKL the default BLAS library. Maybe it is not even worth it to spend time on this at the moment? MKL works out of the box, and it's actually a super-performer. On Mac (and Windows) it requires filling a form on Intel's site to download an installer, and for Linux it's a repo setup for packages, that I've already scripted. Check #3078 https://github.com/kaldi-asr/kaldi/issues/3078 and decide. In any case. I think the best is to see if the push towards MKL will be non-problematic in the end, and then decide whether this issue needs more effort on our side.

ATLAS and OpenBlas packages are borked on debian too, you guessed it absolutely right! atlas-devel comes with the clapack.h file that has #include

in it, but the second file is renamed cblas-atlas.h. so the includes do not even work out of the box. OpenBLAS and cblas-devel packages also have their cblas.h file renamed with a package name prefix. It's a mess. I remember Ubuntu 14.04 packages all placed cblas.h into /usr/include, but all included each other in the "conflicts with" field, so only one could have been installed. At some moment, they changed that by renaming cblas.h files. The only sensible solution I can think of is create a single-line include file named cblas.h (which in turn would include a correctly named file) in a place where there are no other include files (e.g. src/), and tuck that path to the end of compile's include path, so it's looked up for the <> includes the latest (needed to salvage ATLAS debian packaging; we could use the "" includes in our code but that would not help). But again, this may be put now on backburner, in my opinion. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub , or mute the thread .
danpovey commented 5 years ago

Oh wait, I was wrong... currently when we use OpenBLAS we get the headers from OpenBLAS source checked out from github, not the standard ATLAS headers that we include in the repo.

On Sun, Mar 31, 2019 at 1:57 PM Daniel Povey dpovey@gmail.com wrote:

In my opinion it was a mistake to attempt to use the system-provided includes, because the includes that we ship with Kaldi (or download) work just fine. I mean, it was solving a problem that didn't exist, just for the sake of being theoretically the right way to do things.

But I do think it's a good idea to retain the ability to use OpenBLAS; there might be systems where it's faster than MLK, and I don't want our code to become dependent on peculiarities of how MLK works.

On Sun, Mar 31, 2019 at 6:21 AM kkm (aka Kirill Katsnelson) < notifications@github.com> wrote:

@galv https://github.com/galv, we seem to be moving into making MKL the default BLAS library. Maybe it is not even worth it to spend time on this at the moment? MKL works out of the box, and it's actually a super-performer. On Mac (and Windows) it requires filling a form on Intel's site to download an installer, and for Linux it's a repo setup for packages, that I've already scripted. Check #3078 https://github.com/kaldi-asr/kaldi/issues/3078 and decide. In any case. I think the best is to see if the push towards MKL will be non-problematic in the end, and then decide whether this issue needs more effort on our side.

ATLAS and OpenBlas packages are borked on debian too, you guessed it absolutely right! atlas-devel comes with the clapack.h file that has #include

in it, but the second file is renamed cblas-atlas.h. so the includes do not even work out of the box. OpenBLAS and cblas-devel packages also have their cblas.h file renamed with a package name prefix. It's a mess. I remember Ubuntu 14.04 packages all placed cblas.h into /usr/include, but all included each other in the "conflicts with" field, so only one could have been installed. At some moment, they changed that by renaming cblas.h files. The only sensible solution I can think of is create a single-line include file named cblas.h (which in turn would include a correctly named file) in a place where there are no other include files (e.g. src/), and tuck that path to the end of compile's include path, so it's looked up for the <> includes the latest (needed to salvage ATLAS debian packaging; we could use the "" includes in our code but that would not help). But again, this may be put now on backburner, in my opinion. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub , or mute the thread .
kkm000 commented 5 years ago

OpenBLAS still required for Android. So yeah, we need to support it, too.

cdluminate commented 4 years ago

Debian/Ubuntu/Gentoo BLAS/LAPACK maintainer here. These linux distributions support a "BLAS/LAPACK runtime switching" feature which allows users to switch the underlying BLAS/LAPACK implementations without recompiling anything.

https://wiki.gentoo.org/wiki/Blas-lapack-switch https://wiki.debian.org/DebianScience/LinearAlgebraLibraries

That means we generally build packages against the netlib BLAS/LAPACK. And users can choose the exact runtime library they like, e.g. Atlas, OpenBLAS, or even MKL.

I'm about to start investigating how to package kaldi for official debian repo. As a distro maintainer I'd encourage upstream to maintain the support for system provided libraries, because that would make our work much easier.

On the other hand, you mentioned the "somehow" weird openblas package installation layout. I'm afraid that the layout has been significantly changed due to some recent feature updates to Debian.

danpovey commented 4 years ago

Hi, I think it would be the most useful if you could help with PRs to Kaldi itself as regards to our installation on Debian and so on, to see if anything could be improved. Kaldi itself does not lend itself that well to being installed in system space, just the way it's structured and intended to be used. (e.g. getting git updates would be less easy if installed that way). Plus, Kaldi is intended to be a tool for fairly advanced developers, and having to install it themselves (which is not that difficult) is actually a useful filtering step to get rid of people who would not be able to benefit from it and would just waste our time with questions.

cdluminate commented 4 years ago

@danpovey Does that mean it's more recommended to improve the scripts in Kaldi, rather than trying to integrate it into any distro?

danpovey commented 4 years ago

Yeah. To improve how we ingest / deal with the Debian packages.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.