search

celeritas-project / celeritas

Celeritas is a new Monte Carlo transport code designed to accelerate scientific discovery in high energy physics by improving detector simulation throughput and energy efficiency using GPUs.
https://celeritas-project.github.io/celeritas/
Other
65 stars 35 forks source link

Reduce default maximum substeps in field propagator #1384

Closed amandalund closed 2 months ago

amandalund commented 2 months ago

Following up on #1236, this reduces the default maximum number of substeps (per step iteration) in the field propagator from 100 to 10. There's a tradeoff between tracks taking more substeps in fewer step iterations vs. fewer substeps in more step iterations, but the latter significantly improves load blancing in the along-step kernel, and the number of tracks taking many substeps to converge is small (e.g., <2% of tracks take more than 10 total substeps in cms2018). This shows the change in throughput and work for the field regression problems using 10 max substeps vs. 100: rel-throughput-maax-substeps rel-work-max-substeps

I also gathered some information on the number of looping tracks killed before and after this change, and reducing the substep limit didn't seem to have a significant impact:

Problem Max substeps Killed looping tracks Fraction of total tracks Energy deposited [MeV]
testem3-flat+field-orange 10
100		 1
0		 3.36991985e-09
0.00000000e+00		 0.0015739
0
testem3-flat+field+msc-orange 10
100		 11
8		 3.68980412e-08
2.68380684e-08		 8.62542799
12.881228
testem3-flat+field+msc-vecgeom 10
100		 7
5		 2.34880847e-08
1.67727706e-08		 11.822452
4.8461276
testem3-expanded+field+msc-orange 10
100		 5
4		 1.93177693e-08
1.54548461e-08		 1.6958606
1.92713343
testem3-expanded+field+msc-vecgeom 10
100		 6
10		 2.31851377e-08
3.86507106e-08		 10.431617
14.046066
testem3-composite+field+msc-orange 10
100		 9
6		 3.33513164e-08
2.22227110e-08		 9.28268645
7.726298
testem3-composite+field+msc-vecgeom 10
100		 9
4		 3.33327147e-08
1.48183963e-08		 15.615433
2.7987901
cms2018+field+msc-vecgeom 10
100		 115
149		 3.91912153e-07
5.07660725e-07		 735.4356917
961.003735
amandalund commented 2 months ago

It's interesting that this affects orange performance on testem3 quite a bit but not vecgeom. It's not clear to me why we might be hitting the substep limit more often with orange...

sethrj commented 2 months ago

@amandalund one potential is that with v0.5 the auto-converted testem3 is a bit wacky (it has a "background volume" of a cube minus all the enclosing cubes) and could be returning a safety distance of zero when it shouldn't be...