openblas issue with multithreading and make.

[MiguelSen]

hi, all, i'm using a raspberry pi 3, with raspbian 32 bit (armv7). I'm working on a code that basically do svd using lapack function dgesvdx_ (and do other minor functions like fft, hankel matrix), this code works on temporal windows, every window has an input matrix and it simulates a real-time application taking the input from a txt.file. I can compile the code using gcc (version 6.3.0), but the code stops working without any error displayed in the terminal after a few window's processes and the cpu continues to use about the 400% in top application. Using dbg debugger sometimes i receive a segmentation fault or a SIGILL in some random part of the code but the most of the time nothing.This issue doesn't appear if i choose export OMP_NUM_THREADS=2, but it appears with 3 and 4 threads. First i tried the same code with fedora 64 bit (ARMV8) and i don't have this issue (but the performances are worse than 2 threads of raspbian). If i try to make the openblas from source, during the execution of make (make FC=gfortran TARGET=ARMV7) i receive a SEGMENTATION FAULT every time at a different point. Are these issues correlated? Have i to give up at the idea that i can use only 2 threads (and 200% of CPU) in my code using raspbian? Thank you for your answers.

[martin-frbg]

Which version of OpenBLAS are you using (or trying to build) ? There has been some work on fixing thread races in recent months. make segfaulting however makes it sound more like a hardware problem (is it trying to run jobs in parallel perhaps ?). With ARMV8 the speed issue may have been from using a default TARGET=ARMV8 build that is not making use of cpu-specific optimizations, you may be able to specify CORTEXA57 as the cpu target.

[MiguelSen]

i'm trying to build the OpenBLAS-0.2.20. Yes, it's trying to do it in parallel, using top i see 4 process cc1, so the problem is parallelization, right? is there a solution for this? i tried to build the library by source because my code doesn't work with more than 2 threads and i thought that was an issue of the pre-compiled library.

[martin-frbg]

Try setting MAKE_NB_JOBS=2 with make (see Makefile.rule for all available options), maybe your system is overheating when all cores are busy ?

[MiguelSen]

thank you, now i try, but the main problem is that i want that raspberry uses all the 4 cores, so if i can do it with this option and then i can't using all the 4 cores is not very useful for me.

i knew that raspberry pi 3 is using CORTEXA53, or not?

[martin-frbg]

See if make runs to completion when it uses only two cores. Any segmentation faults you saw there will not have much if anything to do with OpenBLAS (unless they happened in the tests at the very end of the build), but with how that particular raspi handles high cpu loads.

[MiguelSen]

Yes, make runs now, thank you!

OS ... Linux Architecture ... arm BINARY ... 32bit C compiler ... GCC (command line : gcc) Fortran compiler ... GFORTRAN (command line : gfortran) Library Name ... libopenblas_armv7p-r0.2.20.a (Multi threaded; Max num-threads is 4)

i'd like to know if it's a issue of my raspberry or a general bug now.

[martin-frbg]

I do not think it is a general bug - as I wrote, make running four compilers in parallel has little or nothing to do with the program it compiles. The program locking up with four threads may be a bug in OpenBLAS, we would need to know the function that hangs (and you could try compiling with USE_SIMPLE_THREADED_LEVEL3=1 if it happens in a blas3 function). Random faults with SIGILL on the other hand make it look more like hardware again, perhaps insufficient cooling or a problematic power supply ? (CORTEXA53 is what the pi3 seems to be using according to various sources on the internet. OpenBLAS has optimized ARMV8 code for the A57 and I believe they both use the same instruction set. I do not have a raspberry pi or any other ARM hardware however.)

[MiguelSen]

i'm using the official power supply of the raspberry, i thought about cooling yesterday so i removed the case and i used an hairdryer (yeah, that's the best i had) to cool the raspberry but it didn't work, using /opt/vc/bin/vcgencmd measure_temp to check the temperature. I have to use USE_SIMPLE_THREADED_LEVEL3=1 in "make" command, right?

[martin-frbg]

Yes, that is a setting for the make command. Do you have a heat sink on the cpu ? Seems these are optional and not included in the standard kit. (If you checked the temperature you will already have an idea how much it changes between idle and maximum load)

[brada4]

What command generates segmentation fault? Is it a compiler or any of the tests?

[MiguelSen]

no, i don't have heat sink, with the case the cpu's temperature goes from 40°-45°to 70°-75° celsius, without case and using the hairdryer remains to a value of about 55°-60° celsius.

@brada4 well, every time changes i have to uninstall the precompiled openblas before to compile the code or is it unnecessary?

[brada4]

You can Google for pi3 overheating solutions. It is not openblas issue. You can heat any processor with one nice yes >/dev/null & per core with same success. Probably fedora got thermal throttling right, but this raspbian let's it heat to visible failure.

Presence of other BLAS libraries does not influence build process: https://github.com/xianyi/OpenBLAS/wiki/faq#debianlts

[MiguelSen]

Probably you're right, it's not a openblas issue but i don't think that it's a temperature's issue. Without case the temperature with 2 cores and 3-4 cores are pretty similar and lower than the temperature with the case and raspberry with 2 core (and it works in that case), but i'll buy the heatsink anyway. Thank you for the information :)

[brada4]

You can close this issue as unrelated to OpenBLAS.

[MiguelSen]

well, is there a section for raspbian/raspberry here?

[brada4]

Nope, there is no section on repairing burned CPUs . Nor a space for comment that numeric processing warms CPU. I don't see how it is OpenBLAS problem that your C compiler crashes at random.

[martin-frbg]

As stated above I am not familiar with the raspberry, but running under maximum load will put strain on other components besides the cpu as well. A quick google found this thread https://www.raspberrypi.org/forums/viewtopic.php?t=147170 which may or may not help.

[brada4]

One can under clock CPU via cpufreq, like set governor to powersave. Or park some of cores permanently. I,d prefer to go with fedora that apparently maintains good thermal envelope.

[brada4]

PS reading link of Martin it seems that extra cooling is necessary. PPS stop heating CPU , it will go in smoke soon under torture

[MiguelSen]

@martin-frbg thank you for the link, so it seems to be an issue with overclocking and frequencies... @brada4 i ordered extracooling, it arrives next week but i don't think that it's a overheating issue because the program stopped also when it was 55°, do you have personal experiencies with fedora?

[martin-frbg]

Maybe that forum I linked to can offer more assistance - but as long as you see problems already when compiling something, OpenBLAS cannot be the cause. Overclocking may cause similar symptoms, but there may be something else wrong with your setup - a bad power supply, or in the worst case a bad board. I did not look around much on that site, but perhaps there are better threads there for finding out if such problems under high load are known and if anything can be done about them.

[MiguelSen]

you're right, thank you @martin-frbg and also @brada4