WIP: Multithreading tests (do not merge)

astrorama / SourceXtractorPlusPlus

SourceXtractor++, the next generation SExtractor

https://astrorama.github.io/SourceXtractorPlusPlus/

GNU Lesser General Public License v3.0

72 stars 9 forks source link

WIP: Multithreading tests (do not merge) #541

Closed marcschefer closed 1 year ago

mkuemmel commented 1 year ago

Is this equivalent to permanently --thread-count 0?? In any case, it doe run. It never uses more than 4 cores: test_thread_off I am assuming this will continue to the end.

marcschefer commented 1 year ago

Is this equivalent to permanently --thread-count 0??

No it's different, this one has the pre-fetcher working normally just the multi-threaded measurement is changed to equivalent of thread-count=0. So I guess we can completely rule out the pre-fetcher as the potential cause.

mkuemmel commented 1 year ago

Okay. So the multi-threading still comes from lower level libraries like levmar or fft3 or so?

If yes, where comes the limitation to n_core=4? There are more cores accessible.

marcschefer commented 1 year ago

Okay. So the multi-threading still comes from lower level libraries like levmar or fft3 or so?

If yes, where comes the limitation to n_core=4? There are more cores accessible.

Good question, since that should not happen I have no idea, my guess is that environment variable is set to 4 by the job system.

mkuemmel commented 1 year ago

The machine I am running has not scheduler. This was discussed on the ticket today. I give still --thread-count 4, maybe this is somehow forwarded? But even when I set --thread-count 0 (on develop) it sticks to 4 cores.

mkuemmel commented 1 year ago

I ran the test version three times within the pipeline, asking 4, 6 and 8 cores. They all run and the # of cores shows in the profile: test_pipeline_4 test_pipeline_6 test_pipeline_8

But the computing time is always the same (248-256min), so the # of cores does not really help. No idea what this means.

On Friday Simon ran with strace attached to it. Unfortunately he got a circular run, so it did not help. But he'll try again.

mkuemmel commented 1 year ago

My argument from this morning is:

the profiling figure above show three runs of the identical SE++ version (multi-threading switched off);
in the runs different numbers of core were available;
the figure shows that actually all available core were used;
nevertheless the runtime is basically identical;

In total it is not clear to me whether the additional core that were used did something useful. It would be interesting to compare this to the runtime at SDC-CH, which was multi-threaded.

marcschefer commented 1 year ago

Closing this, since the bug has been found