search

carstenbauer / ThreadPinning.jl

Readily pin Julia threads to CPU-threads
https://carstenbauer.github.io/ThreadPinning.jl/
MIT License
106 stars 7 forks source link

Pinning affects an external OpenMP library #94

Closed PetrKryslUCSD closed 4 months ago

PetrKryslUCSD commented 4 months ago

TLDR: An external OpenMP-equipped C++ library shows no parallel speedup when thread pinning is used. Disabling pinning fixed the issue.

https://discourse.julialang.org/t/openmp-in-a-shared-library-opened-in-julia/113880/9

carstenbauer commented 4 months ago

Are you perhaps oversubscribing CPU cores? That is, are Julia threads and the external OpenMP threads maybe running on the same set of CPU cores? This would explain why it works better without thread pinning, because the OS scheduler then has the flexibility to avoid this (as much as possible).

If that's the case you should either avoid thread pinning (as you did) or pin the OpenMP threads as well (e.g. through environment variables) - to different cores, of course.

Otherwise, pinthreads shouldn't effect any OpenMP threads: Julia threads aren't even OpenMP threads, they are pthreads.

PetrKryslUCSD commented 4 months ago

I do not think the cpu cores are oversubscribed. When the library is called, only a single thread is running.

carstenbauer commented 4 months ago

I don't understand, I thought you have multiple threads? Why would pinning or not pinning make a big/any difference if there is only a single thread running?

In any case, pinthreads doesn't change anything about other OpenMP threads. Hence, AFAICS, this isn't a bug in ThreadPinning.jl.

(Also, you neither provide much information nor a MWE, which makes this hard/impossible to figure out.)

PetrKryslUCSD commented 4 months ago

The issue was intended as a heads up. I'm working on a minimal working example.

The application runs with multiple threads. However at the point where the library gets called, only a single thread is running. The issue is real: enabling or disabling pin threading affects the performance of the external library.

carstenbauer commented 4 months ago

The issue is real: enabling or disabling pin threading affects the performance of the external library.

To be clear, I'm not arguing that your performance issue isn't real. I'm saying that it (almost certainly) isn't a bug in ThreadPinning.jl.

Of course, feel free to continue the discussion in the Discourse thread. However, much more information will be needed, like:

PetrKryslUCSD commented 4 months ago

Here is the minimal working example (https://www.dropbox.com/scl/fi/3dezt0lp10pdflcq9479z/mwe_eclgraphcolor.tgz?rlkey=14w67j59xz3uk3fczergzhhmu&dl=0).

Here is a sample output:


pkrysl@horntail:~/mwe_eclgraphcolor$ . ./run_mwe.sh
================================================================================
With 1 thread
================================================================================
  Activating project at `~/mwe_eclgraphcolor`
[ Info: Loading ThreadPinning
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Updating git-repo `https://github.com/PetrKryslUCSD/ECLGraphColor.jl.git`
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Building ECLGraphColor → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/28351fdf0c4c6de0974f0f560c914d6bfd9411dc/build.log`
  0.560385 seconds
  Activating project at `~/mwe_eclgraphcolor`
[ Info: NO ThreadPinning
    Updating git-repo `https://github.com/PetrKryslUCSD/ECLGraphColor.jl.git`
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Building ECLGraphColor → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/28351fdf0c4c6de0974f0f560c914d6bfd9411dc/build.log`
  0.498791 seconds
================================================================================
With 2 threads
================================================================================
  Activating project at `~/mwe_eclgraphcolor`
[ Info: Loading ThreadPinning
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Updating git-repo `https://github.com/PetrKryslUCSD/ECLGraphColor.jl.git`
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Building ECLGraphColor → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/28351fdf0c4c6de0974f0f560c914d6bfd9411dc/build.log`
  0.564154 seconds
  Activating project at `~/mwe_eclgraphcolor`
[ Info: NO ThreadPinning
    Updating git-repo `https://github.com/PetrKryslUCSD/ECLGraphColor.jl.git`
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Building ECLGraphColor → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/28351fdf0c4c6de0974f0f560c914d6bfd9411dc/build.log`
  0.280195 seconds
================================================================================
With 4 threads
================================================================================
  Activating project at `~/mwe_eclgraphcolor`
[ Info: Loading ThreadPinning
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Updating git-repo `https://github.com/PetrKryslUCSD/ECLGraphColor.jl.git`
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Building ECLGraphColor → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/28351fdf0c4c6de0974f0f560c914d6bfd9411dc/build.log`
  0.595618 seconds
  Activating project at `~/mwe_eclgraphcolor`
[ Info: NO ThreadPinning
    Updating git-repo `https://github.com/PetrKryslUCSD/ECLGraphColor.jl.git`
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Building ECLGraphColor → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/28351fdf0c4c6de0974f0f560c914d6bfd9411dc/build.log`
  0.155279 seconds
pkrysl@horntail:~/mwe_eclgraphcolor$

Two Julia files are provided: one adds, uses ThreadPinning (mwe_w.jl). This one does not get any speed up on the graph calculation.

The other one does (mwe_wo.jl).

PetrKryslUCSD commented 4 months ago

@carstenbauer Carsten, I am not saying that it is a bug in ThreadPinning. It would be interesting to find out why thread pinning affects OpenMP threads, though. Wouldn't you agree?