astrorama / SourceXtractorPlusPlus

SourceXtractor++, the next generation SExtractor
https://astrorama.github.io/SourceXtractorPlusPlus/
GNU Lesser General Public License v3.0
72 stars 9 forks source link

Investigate SE++ scaling with multi-threading #477

Open mkuemmel opened 2 years ago

mkuemmel commented 2 years ago

I ran some Euclid test data with single image fitting (3 bands, ~4 images per band per pixels) using different number of cores to test the scaling of the multi-threading. This investigation is similar to #411, but some more care was taken to only use physical cores (no hyperthreading). The system information is:

There are 64 physical cores. I ran it with different # of physical cores which results to (the basic data is below): scaling_SEPP

Note that I did use hyperthreading for the last data point wit 124 cores. I repeated the measurements for many core numbers (with days in between the runs) but finding virtually identical run times.

# 52197 objects
# ncore  time_nohyper time_nh
  4      38:50:07     38.835
  8      20:07:38     20.127
  8      20:13:18     20.222
 12      13:21:53     13.365
 16      10:30:01     10.500
 16      10:31:01     10.500
 20      09:56:07      9.935
 24      08:18:02      8.300
 24      08:16:05      8.268
 28      06:29:01      6.484
 32      05:41:24      5.690
 32      05:42:07      5.702
 36      05:19:22      5.323
 40      05:08:39      5.144
 40      05:05:54      5.098
 44      04:25:53      4.432
 48      04:01:26      4.023
 48      04:04:29      4.074
 52      03:53:10      3.886
 56      03:49:48      3.830
 56      03:49:02      3.817
 60      03:27:04      3.451
 64      03:24:02      3.400
 64      03:24:32      3.409
128      03:00:22      3.006