LonelyCat124
commented
1 year ago Tracer Advection + vtune:
Problem size:
This seems to primarily be memory bound - particularly with cross NUMA accesses being a significant amount of memory accesses. I'll show results here with both. I disabled the checksum and IO routine at the end of the simulation as it throws off the results and massively warps the results.
export OMP_PLACES=cores
export OMP_PROC_BIND=close/spreadclose proc bind:
A couple of things I note for this:
NUMA remote accesses are very high (66%)
Parallel region time is good (99.4%) - mostly with high physical core utilisation
I compiled this example with -xCORE_AVX512 but vectorisation is still only 256-bit, not sure why.
CPI is 7.843, which is pretty bad since in theory I think we could be looking at <1.
spread proc bind:
It looks like either way the NUMA behaviour is not good, which may be why we're seeing only 256-bit vectorisation as maybe the compiler doesn't see it as worthwhile. I will try to do a memory analysis with close and see what we get.
Looking at the memory analysis it seems like loops that access the mydomain array have higher average latency (in cycles) than other loops, though its not clear why that might be.
The worst offender is:
!$omp parallel default(shared), private(idx_12,idx_13,jk)
!$omp do schedule(auto)
do jk = 2, jpk - 1, 1
do idx_12 = LBOUND(zwx, 2), UBOUND(zwx, 2), 1
do idx_13 = LBOUND(zwx, 1), UBOUND(zwx, 1), 1
zwx(idx_13,idx_12,jk) = tmask(idx_13,idx_12,jk) * (mydomain(idx_13,idx_12,jk - 1) - mydomain(idx_13,idx_12,jk))
enddo
enddo
enddo
!$omp end do
!$omp end parallelThe reported average latency in cycles in this loops is 1288 (the best loop has an average of 167). The next worse has 1030 average:
!$omp parallel default(shared), private(ji,jj,jk,ztra)
!$omp do schedule(auto)
do jk = 1, jpk - 1, 1
do jj = 2, jpj - 1, 1
do ji = 2, jpi - 1, 1
ztra = -zbtr * (zwx(ji,jj,jk) - zwx(ji,jj,jk + 1))
mydomain(ji,jj,jk) = ztra
enddo
enddo
enddo
!$omp end do
!$omp end parallelBoth loops have relatively low arithmetic intensity, but are by no means the lowest.
I'm still not clear why we have 63% of remote accesses - the arrays are sensibly initialised, so the only option is if schedule(auto) does something unusual. I will try to chance the schedule to a static schedule and see how that affects the memory access patterns.
Switching to schedule(auto) completely changes the memory analysis. Cross NUMA memory accesses drop to almost 0:
LLC miss rate increases slightly, but the worst loop's average latency drops by almost half to 759. I'm going to rerun the base analysis now to see how the CPI changes from this too.
CPI down to 5.005 with this change, I'll see how the actual runtime changed.
Runtime with static:
Thread 0
Initialisation 1 0.31572E+00 0.31572E+00 0.31572E+00 0.00E+00
Time-stepping 1 0.90016E+02 0.90016E+02 0.18003E+00 0.00E+00Runtime with auto:
Thread 0
Initialisation 1 0.32153E+00 0.32153E+00 0.32153E+00 0.00E+00
Time-stepping 1 0.13415E+03 0.13415E+03 0.26830E+00 0.00E+00Looks like swapping from auto to static with intel compiler is a 40% runtime improvement for this case @sergisiso
I'm going to see if I can generate a roofline next.
Tracer advection roofline:
Matches exactly what vtune shows - the tracer advection benchmark is DRAM limited - the main performance improvement would be finding a way to improve cache efficiency or higher arithmetic intensity, but given the nature of the problem it seems unlikely.
The largest nodes are:
- The parallel region at 226 (starting with
z0u = SIGN(0.5d0, pun(ji,jj,jk))) - The parallel region at 343 (starting with
z0w = SIGN(0.5d0, pwn(ji,jj,jk + 1))) - this one shows some gap in the roofline between the performance and the roofline too. - The parallel region at 170 (pretty small loop in terms of work).
The 2nd loop is the one with a peel and remainder loop, hard for PSyclone to do much about that though I expect for now.
sergisiso
arporter
I figured I'd make an issue to detail results with different tools we're using as part of the PA workshop.
Probably we'll look at both NemoLite (incl distributed memory) and tracer advection.
First tool is Vtune, and i'll post (and update) things as I get results. @vkatkinson If you also have results to share it would be good to place here as well probably.