Closed KoykL closed 2 years ago
Hi KoykL,
thank for the report. In order to reproduce the problem, can you give me some information about your system. CPU, OS, Compiler? OpenMP Library?
OS: Fedora 35 Compiler: gcc 11.2.1 openmp runtime: libgomp-11.2.1 flexiblas: 3.0.4 cpu: epyc 7502p
So I tried to reproduce the bug on the following system:
USE_OPENMP=1 USE_SHARED=1
)I compiled FlexiBLAS with
cmake3 ../ -DDEV=ON -DEXTRA="OB19" -DOB19_LIBRARY="/scratch/openblas-0.3.19/lib/libopenblas.so;gomp;pthread"
make -j 16
and then executed a DGEMM
benchmark with FlexiBLAS-Openblas and OpenBlas linked directly. (The benchmark tool can be found in the examples
folder after build)
# DGEMM Benchmark on all 128 Cores ( 64 + 64 HT) for two 20000x20000 matrices using only the OB19 backend(OpenBLAS 0.3.19 from above)
$ OMP_NUM_THREADS=128 OMP_PROC_BIND=true ./benchmark -b DGEMM -d 20000 -r 1 -o OB19
# Dimension: 20000
# Runs: 1
# Benchmark: DGEMM
# Only: OB19
# Name Runtime GFlops
OB19 1.70541580e+01 9.38187627e+02
# Benchmark with directly linked OpenBLAS 0.3.19:
$ OMP_NUM_THREADS=128 OMP_PROC_BIND=true ./benchmark.OB19 -b DGEMM -d 20000 -r 1
# Dimension: 20000
# Runs: 1
# Benchmark: DGEMM
# Name Runtime GFlops
OB19 1.70563250e+01 9.38068431e+02
with are up to small disturbances the same performance values. I also monitored it in background with top which CPU cores are utilizes and all 128 cores are working in both cases.
Thus, I could not reproduce your problem. Please provide me a self containing MWE example, which I can build and debug. Otherwise I close this issue within the next days.
@KoykL Are you using FlexiBLAS as provided in the official Fedora repos? Cannot reproduce here either.
I am using official repo flexiblas. However, I am calling blas from python (numpy). Let me investigate a bit more on what is actually happening.
I figured out why you cannot reproduce the problem.
The compiler arguments for building benchmark by default includes "-fopenmp". Including openmp runtime in the main program probably causes openmp runtime not to be restarted across the two dlopen call in __flexiblas_dlopen.
To reproduce the behavior I reported, remove "-fopenmp" for benchmark.c. e.g.
Original command:
/usr/bin/cc -fPIC -std=c99 -D_FILE_OFFSET_BITS=64 -fopenmp -Wno-unused-parameter -O3 -DNDEBUG -rdynamic CMakeFiles/benchmark.dir/benchmark.c.o -o benchmark -Wl,-rpath,/tmp/flexiblas/build/lib -lm -ldl ../lib/libflexiblas.so.3.0 ../libcscutils/lib/libcscutils.a -lm -ldl -lgfortran -lm -lgfortran -lm -lquadmath -lm
Modified command:
/usr/bin/cc -fPIC -std=c99 -D_FILE_OFFSET_BITS=64 -Wno-unused-parameter -O3 -DNDEBUG -rdynamic CMakeFiles/benchmark.dir/benchmark.c.o -o benchmark -Wl,-rpath,/tmp/flexiblas/build/lib -lm -ldl ../lib/libflexiblas.so.3.0 ../libcscutils/lib/libcscutils.a -lm -ldl -lgfortran -lm -lgfortran -lm -lquadmath -lm
If successful, when you run benchmark with additional environmental variable OMP_DISPLAY_ENV=VERBOSE, you will see "OPENMP DISPLAY ENVIRONMENT BEGIN" twice, indicating openmp runtime has been restarted, and triggering the bug I originally reported.
With the original compiler arguments, "OPENMP DISPLAY ENVIRONMENT BEGIN" will only be printed once.
I confirm that the bug exists at least for the stock version of numpy. E.g.:
$ docker run --rm -it fedora:35
$ dnf -y install python3-numpy wget
$ wget https://raw.githubusercontent.com/dmytrov/benchmark-GEMM/master/numpy-benchmark.py
$ python3 numpy-benchmark.py
I see all my cores spinning. However,
$ OMP_PROC_BIND=true python3 numpy-benchmark.py
spins a single core. I tried GEMM benchmarking in R with and without OMP_PROC_BIND=true
and there's no issue. Notably, R includes -fopenmp
:
$ R CMD config --ldflags
-Wl,--export-dynamic -fopenmp -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -L/usr/lib64/R/lib -lR -ltre -lpcre2-8 -llzma -lbz2 -lz -lrt -ldl -lm -licuuc -licui18n
Numpy does not.
I could reproduce the error now myself. Searching for the problem I found that serval other projects interfacing OpenMP from Python had similar problems. I found two different solutions for the problem (at the moment):
I would implement it in the following way:
If this behavior is fine, I include it in the next release (@Enchufa2 scheduled for tomorrow)
I think this is a good solution to the problem.
Sounds good to me. :)
Closed with 3.1.0
When loading the openblas-openmp library in flexiblas.c, __flexiblas_dlopen is used, which calls dlopen on openblas-openmp twice. The first call is used to retrieve global variables such as "flexiblas_ld_global", and the value is used to make the second "actual" dlopen call.
However, when OMP_PROC_BIND is used, during the first dlopen call to openblas-openmp, openmp runtime will set the cpu affinity mask of the current thread to just the first cpu core. During the second dlopen call to openblas-openmp, this affinity mask will be picked up by the openmp runtime, and make the openmp runtime to think that the first cpu core is the only available cpu. Therefore, only the first cpu will be used when OMP_PROC_BIND is used with openblas-openmp and flexiblas.
Is there a way to use all cpu cores when cpu binding is used with openmp? Thank you!