Excluding the issues I identified in #237 and that have been partially fixed in #239. I've found that the Sombrero ITT-sn benchmark is too small for the largest nodes (in terms of number of cores). In particular, on the CSD3 Sapphire Rapids, with 112 cores per node (although I am only using 108 tasks due to #237 (comment)), the results can fluctuate massively between repeated runs (even on the exact same node) - I have observed variations as much as ~80%.
Changing the arguments passed to the executable from -s small to -s medium resolve this problem for me but this is currently what is used for the 64 node benchmark. I don't have the resources to test this but I suspect this benchmark will also be too small for 64 nodes.
giordano
commented
9 months ago
Excluding the issues I identified in #237 and that have been partially fixed in #239. I've found that the Sombrero ITT-sn benchmark is too small for the largest nodes (in terms of number of cores). In particular, on the CSD3 Sapphire Rapids, with 112 cores per node (although I am only using 108 tasks due to #237 (comment)), the results can fluctuate massively between repeated runs (even on the exact same node) - I have observed variations as much as ~80%.
Changing the arguments passed to the executable from
-s smallto-s mediumresolve this problem for me but this is currently what is used for the 64 node benchmark. I don't have the resources to test this but I suspect this benchmark will also be too small for 64 nodes.