search

open-ce / pytorch-feedstock

Apache License 2.0
5 stars 11 forks source link

OpenBLAS Warning : Detect OpenMP Loop and this application may hang. Please rebuild the library with USE_OPENMP=1 option. #34

Closed cdeepali closed 3 years ago

cdeepali commented 3 years ago

Executing test_jit tests with OpenCE is showing numerous warnings:

OpenBLAS Warning : Detect OpenMP Loop and this application may hang. Please rebuild the library with USE_OPENMP=1 option.

This same warning is not observed with WMLCE for same version of OpenBLAS.

cdeepali commented 3 years ago

As a workaround user can set OMP_NUM_THREADS=1 to avoid using multiple threads.

cdeepali commented 3 years ago

https://github.com/pytorch/pytorch/issues/52047

lgg commented 3 years ago

@cdeepali i had the same issue. Building OpenBLAS from source with USE_OPENMP=1 didn't help for me.

I have few Jetson Nano and I guess i found a few ways to fix it:

  1. Downgrading Jetpack version On Jetson Nano with JetPack 4.4.1 - everything works fine, on new Jeston Nano with JetPack 4.5.1 - i have the same issue. Check the difference from jtop output:

Jetpack 4.4.1 image

Jetpack 4.5.1 image

Nvidia update VPI (https://docs.nvidia.com/vpi/index.html) and i guess it cause the problem.

  1. Downgrading torch On another Jetson Nano with Jetpack 4.5.1 i tried this:
    • sudo apt-get install libopenblas-base libopenmpi-dev
    • wget https://nvidia.box.com/shared/static/wa34qwrwtk9njtyarwt5nvo6imenfy26.whl -O torch-1.7.0-cp36-cp36m-linux_aarch64.whl
    • ./venv/bin/pip3 install torch-1.7.0-cp36-cp36m-linux_aarch64.whl
    • and it helps
bnemanich commented 3 years ago

This was resolved by switching PyTorch to use TBB instead of OpenMP.

lgg commented 3 years ago

@bnemanich how did you do that?

cdeepali commented 3 years ago

We switched to using TBB for intra-op parallelism. - https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html#:~:text=PyTorch%20uses%20a%20single%20thread,(%20intra%2Dop%20parallelism%20).

jayfurmanek commented 3 years ago

yeah, the change was in: https://github.com/open-ce/pytorch-feedstock/commit/2ab85700a93d382a3a67a334a38a2ed29e898736

jayfurmanek commented 3 years ago

And just a note for why this was needed.

Anaconda's toolchain uses gomp for openMP. And gomp is known to not be fork-safe (it can hang when the program using it forks). To avoid any hangs, Anaconda has disabled openMP in their OpenBLAS package. (conda-forge has since included llvm-openmp as a openmp variant, but defaults hasn't moved there yet)

This warning message comes when Pytorch is built with OpenMP support, but then uses an OpenBLAS that was not.

Switching Pytorch to use TBB instead of OMP avoids the problem.

The downside is still relying on the OpenBLAS package from defaults with no OMP support, which is slower. In a future release we will likely include our own OpenBLAS package built with llvm-openmp unless defaults provides.