OpenBLAS dev is not working on E7 Haswell & Broadwell CPUs

[JohnCenteno]

Dear Xianyi,

Do you know why I can not complie OpenBLAS dev"gmake TARGET=HASWELL", but worked with "gamke"? On OpenBLAS 0.2.18 show different message. Thanks. John Centeno gmake TARGET=HASWELL(OpenBLAS dev) ../kernel/x86_64/dtrmm_kernel_4x8_haswell.c:77: Error: no such instruction: vpe rmpd $0x1b,%ymm3,%ymm3' ../kernel/x86_64/dtrmm_kernel_4x8_haswell.c:78: Error: no such instruction:vpe rmpd $0xb1,%ymm2,%ymm2' ../kernel/x86_64/dtrmm_kernel_4x8_haswell.c:79: Error: no such instruction: vpe rmpd $0xb1,%ymm3,%ymm3' gmake[1]: *** [dtrmm_kernel_LN.o] Error 1 gmake[1]: Leaving directory/root/OpenBLAS/kernel' gmake: *\ [libs] Error 1

gmake TARGET=HASWELL(OpenBLAS 0.2.18)

gmake edhat-linux/4.4.7/../../.. -lgfortran -lm -lgfortran -lm -lc && echo OK. OK. rm -f linktest gmake[1]: Leaving directory `/root/OpenBLAS/exports'

OpenBLAS build complete. (BLAS CBLAS LAPACK LAPACKE)

OS ... Linux Architecture ... x86_64 BINARY ... 64bit C compiler ... GCC (command line : gcc) Fortran compiler ... GFORTRAN (command line : gfortran) Library Name ... libopenblas_sandybridgep-r0.2.19.dev.a (Multi threaded; Max num-threads is 60)

To install the library, you can run "make PREFIX=/path/to/your/installation install".

[brada4]

Please read FAQ item about enterprise linux 6. Your assembler does not support AVX2. If you cannot acquire functional binutils you can disable AVX2 for build

[JohnCenteno]

Dear Andrews,

Thanks, but GCC in RHEL 6.6 is 4.7 and different parameters show different result. My CPUs are 4 PCs E7-8855 V4(Broadwell).

gmake TARGET=HASWELL rmpd $0xb1,%ymm2,%ymm2' ../kernel/x86_64/dtrmm_kernel_4x8_haswell.c:79: Error: no such instruction: vpermpd $0xb1,%ymm3,%ymm3' gmake[1]: *** [dtrmm_kernel_LN.o] Error 1 gmake[1]: *** [dtrmm_kernel_LT.o] Error 1 gmake[1]: Leaving directory/root/OpenBLAS/kernel' gmake: *\ [libs] Error 1

gmake TARGET=HASWELL NO_AVX=1 ../kernel/x86_64/dtrmm_kernel_4x8_haswell.c:79: Error: no such instruction: vpermpd $0xb1,%ymm3,%ymm3' gmake[1]: *** [dtrmm_kernel_LT.o] Error 1 gmake[1]: *** [dtrmm_kernel_LN.o] Error 1 gmake[1]: Leaving directory/root/OpenBLAS/kernel' gmake: *\ [libs] Error 1

gmake(sandybridge) gmake NO_AVX=1(Nehalem) OpenBLAS build complete. (BLAS CBLAS LAPACK LAPACKE)

OS ... Linux Architecture ... x86_64 BINARY ... 64bit C compiler ... GCC (command line : gcc) Fortran compiler ... GFORTRAN (command line : gfortran) Library Name ... libopenblas_nehalemp-r0.2.18.a (Multi threaded; Max num-t hreads is 60) gmake(Sandybridge)

[brada4]

Direct link to fix your system: https://github.com/xianyi/OpenBLAS/wiki/faq#binutils

[JohnCenteno]

Dear Andrew,

Thanks. I tried both methods, but I can not install devtools-3 or devtools -4 64bit on my Centos 7 or RHE6 64bit. I only install devtools-2 successfully before on Scientific Linux when I want to find a way to upgrade GCC 4.4.7.

Thanks. John Centos

1. Install a package with repository for your system:

On CentOS, install package centos-release-scl available in CentOS repository:

$ sudo yum install centos-release-scl

On RHEL, enable RHSCL repository for you system:

$ sudo yum-config-manager --enable rhel-server-rhscl-7-rpms

2. Install the collection:

$ sudo yum install devtoolset-3

3. Start using software collections:

$ scl enable devtoolset-3 bash

[brada4]

You dont need to modify CentOS7 in any way. It knows AVX2 already For undecided Scientific RedCent 6 version direct link to binutils package is here: http://mirror.centos.org/centos/6/sclo/x86_64/rh/devtoolset-4/devtoolset-4-binutils-2.25-10.el6.x86_64.rpm Install it with YUM to pull dependencies down the way.

[JohnCenteno]

Dear Andrew,

Thanks. I tried, but it sill not working. John Centeno

No package devtoolset-4 available. Error: Nothing to do [root@localhost ~]# sudo yum install devtoolset-3 Loaded plugins: product-id, refresh-packagekit, security, subscription-manager This system is not registered to Red Hat Subscription Management. You can use su bscription-manager to register. Setting up Install Process Repository rhscl-centos-release-scl is listed more than once in the configuratio n No package devtoolset-3 available. Error: Nothing to do [root@localhost ~]# sudo yum install devtoolset-2 Loaded plugins: product-id, refresh-packagekit, security, subscription-manager This system is not registered to Red Hat Subscription Management. You can use su bscription-manager to register. Setting up Install Process Repository rhscl-centos-release-scl is listed more than once in the configuratio n No package devtoolset-2 available. Error: Nothing to do [root@localhost ~]# sudo yum-config-manager --enable rhel-server-rhscl-7-rpms Loaded plugins: product-id, refresh-packagekit Repository rhscl-centos-release-scl is listed more than once in the configuratio n [root@localhost ~]# sudo yum-config-manager --enable rhel-server-rhscl-7-rpms Loaded plugins: product-id, refresh-packagekit Repository rhscl-centos-release-scl is listed more than once in the configuratio n [root@localhost ~]# sudo yum install devtoolset-3 Loaded plugins: product-id, refresh-packagekit, security, subscription-manager This system is not registered to Red Hat Subscription Management. You can use su bscription-manager to register. Setting up Install Process Repository rhscl-centos-release-scl is listed more than once in the configuratio n No package devtoolset-3 available.

[brada4]

First refresh repo files yum upgrade centos-release Install brand new scl repo yum install centos-release-scl-rh Install just binutils (will not hurt if you install entire gcc5 but it is not reqired for AVX2 support) yum install devtoolset-4-binutils

[JohnCenteno]

Dear Andrew,

Thanks, but I still got error messages.

With Best Regards. John Centeno

Installation

To set up the repository, install the webtatic-release RPM:

Webtatic EL7 for CentOS/RHEL 7:

rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm warning: /var/tmp/rpm-tmp.pVO7yd: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY Preparing... ################################# [100%] package epel-release-7-6.noarch is already installed [root@localhost ~]#

rpm -Uvh https://mirror.webtatic.com/yum/el7/webtatic-release.rpm

https://access.redhat.com/articles/1320623

If above article doesn't help to resolve this issue please open a ticket with Re d Hat Support.

One of the configured repositories failed (testing 2 devtools for CentOS 7Serve r), and yum doesn't have enough cached data to continue. At this point the only safe thing yum can do is fail. There are a few ways to work "fix" this:

 1. Contact the upstream for the repository and get them to fix the problem.

 2. Reconfigure the baseurl/etc. for the repository, to point to a working
    upstream. This is most often useful if you are using a newer
    distribution release than is supported by the repository (and the
    packages for the previous distribution release still work).

 3. Disable the repository, so yum won't use it by default. Yum will then
    just ignore the repository until you permanently enable it again or use
    --enablerepo for temporary usage:

        yum-config-manager --disable testing-devtools-2-centos-7Server

 4. Configure the failing repository to be skipped, if it is unavailable.
    Note that yum will try to contact the repo. when it runs most commands,
    so will have to try and fail each time (and thus. yum will be be much
    slower). If it is a very temporary problem though, this is often a nice
    compromise:

        yum-config-manager --save --setopt=testing-devtools-2-centos-7Server                                                                                                             .skip_if_unavailable=true

failure: repodata/repomd.xml from testing-devtools-2-centos-7Server: [Errno 256] No more mirrors to try. http://people.centos.org/tru/devtools-2/7Server/x86_64/RPMS/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found

[brada4]

CentOS 7 will compile OpenBLAS Haswell kernels out of the box. You need ONE SINGLE additional package to build them on CentOS 6

I dont understand where you dig up all those broken repositories. Please clean them from your system and try to repat FAQ entry to the letter on a system where it applies.

[JohnCenteno]

Dear Andrew,

Thanks. On Centos 7, I have to install GCC 4.8 first and it showed Haswell CPUs. But I can not run "Yum Install development tool". I have to run something to execute "Yum install". By the way, when it will support Broadwell CPUs. My CPUs on this system is E7-4880 V2, but I have the other system is E7-8855 V4.

With Best Regards. John Centono

251 yum-config-manager --disable RHEL_7_Disc 252 yum install "Development Tools" 253 yum-config-manager --disable testing-1.1-devtools-7Server 254 yum install "Development Tools" 255 yum-config-manager --disable testing-devtools-2-centos-7Server 256 yum install "Development Tools" 257 yum install centos-release-scl 258 sudo yum-config-manager --enable rhel-server-rhscl-7-rpms

This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. OpenBLAS build complete. (BLAS CBLAS)

OS ... Linux Architecture ... x86_64 BINARY ... 64bit C compiler ... GCC (command line : gcc) Library Name ... libopenblas_haswellp-r0.2.18.a (Multi threaded; Max num-threads is 80)

To install the library, you can run "make PREFIX=/path/to/your/installation install".

[JohnCenteno]

Dear Andrew,

I still got error message when I compile hpl-2.2. Below are my procedure. Thanks again for your patience and time.

With Best Regards. John Centos

Partil content of MAKE_LINUX_PII_CBLAS LAdir = $(HOME)/OpenBLAS LAinc = LAlib = $(LAdir)/libopenblas.a

OpenBLAS gmake NO_AVX=1 make PREFIX=/rootOpenBLAS install MPICH ./configure --prefix=/root/root/mpich HPL make arch=Linux_PII_CBLAS

make[2]: Entering directory `/root/hpl-2.2/testing/ptest/Linux_PII_CBLAS' /root/mpich/bin/mpif77 -DHPL_CALL_CBLAS -I/root/hpl-2.2/include -I/root/hpl-2.2 /include/Linux_PII_CBLAS -I/root/mpich/include -fomit-frame-pointer -O3 -funrol l-loops -o /root/hpl-2.2/bin/Linux_PII_CBLAS/xhpl HPL_pddriver.o HPL_pdi nfo.o HPL_pdtest.o /root/hpl-2.2/lib/Linux_PII_CBLAS/libhpl.a /root/O penBLAS/libopenblas.a /root/mpich/lib/libmpich.so /usr/bin/ld: /root/OpenBLAS/libopenblas.a(blas_server.o): undefined reference to symbol 'pthread_create@@GLIBC_2.2.5' /usr/bin/ld: note: 'pthread_create@@GLIBC_2.2.5' is defined in DSO /lib64

/libpth read.so.0 so try adding it to the linker command line /lib64/libpthread.so.0: could not read symbols: Invalid operation collect2: error: ld returned 1 exit status make[2]: * [dexe.grd] Error 1 make[2]: Leaving directory `/root/hpl-2.2/testing/ptest/Linux_PII_CBLAS' make[1]: * [build_tst] Error 2 make[1]: Leaving directory`/root/hpl-2.2' make: *\ [build] Error 2

[brada4]

I am astonished how you managed to skip supported Red Hat 6 devtoolset 4.1 which is included even in their 50$/year subscription. I am sorry but your haswell build works like charm, there is nothing to add to this non-issue.

OpenBLAS FAQ deals with static linking issues too. It should not take more than 5 minutes to read to the point where pthreads are mentioned.

[JohnCenteno]

Dear Andrew,

Thanks a lot. Due to the system is for test only for few days(I did not know if I keep reinstall RHEL how to register it), I will format it to reinstall the other OS like SuSe or Windows

I did read, but it kind of difficult for me to understand.

With Best Regards. John Centeno

[brada4]

Red Hat is subscription service. If you dont subscribe ($$$) it does not work. Install CentOS7(Or Scientific Linux 7 if that flatters you better) or OpenSUSE 42 (or fedora or mageia for that sake) and get OpenBLAS 0.2.18 (prebuilt well-followed latest release with all haswell support) from provided repositories. Windows or Debian or Ubuntu will force you back into painful recompilation track if you are not fine with old releases. And there is no need for HPL cluster scheduler on a single system ever.

[JohnCenteno]

Dear Andrew,

My co-workes told me he fixed the problem by modifying HPL compiler file "Make.Linux_PIII_CBLAS". He did not change my configure on OpenBLAS

below 3 items in Make.Linux_PII_CBLAS HPL_INCLUDES = HPL_LIBS = LINKFLAGS =

With Best Regards. John Centeno

[brada4]

Congrats. You can close this non-issue about your broken system that has nothing to do with hpl or openblas.