I realised that here we're writing 3 field sized arrays into main memory unnecessarily. It is potentially increasing the runtime %20.
In the second phase of the algorithm here we pass a part of the du, dud, and d2u into der_univ_subs, and they're all rewritten in place. Then later we combine them in rhs for the final result. Ideally, we want du, dud, and d2u to be read once and rhs to be written only once. However because of the way der_univ_subs work, the updated data in du arrays after der_univ_subs call gets written in the main memory, even though we don't need this at all.
There are three ways we can fix this
In the parallel do loop in the second phase we can copy the relevant parts of du, dud, and d2u arrays into (SZ, n) sized temporary arrays. Then we pass temporary arrays into der_univ_subs, and at the end we use these temporaries to obtain final rhs. This is the easiest solution but it may not be the best in terms of performance.
We can write an alternative der_univ_subs and separate input and output arrays. This way we can pass a part of the du arrays as we do now, and pass a small temporary array as the output one. Because du arrays will be input arrays no data will be written in main memory. Then we can combine the temporaries to get rhs.
If we're writing an alternative der_univ_subs to be used in transeq, we can go one step further and have a fused version of it. This would probably the most performant solution. der_univ_subs is relatively lightweight so it isn't really hard to do so. The new subrotuine can input all du, dud, and d2u, and write the final result rhs.
I'm copying my comment in #27 so that we don't forget about it.
https://github.com/xcompact3d/x3d2/blob/2d906a5cafe6060b8021b66aea8ff360558c3968/src/omp/exec_dist.f90#L164-L185
I realised that here we're writing 3 field sized arrays into main memory unnecessarily. It is potentially increasing the runtime %20.
In the second phase of the algorithm here we pass a part of the
du,dud, andd2uinto der_univ_subs, and they're all rewritten in place. Then later we combine them inrhsfor the final result. Ideally, we wantdu,dud, andd2uto be read once andrhsto be written only once. However because of the way der_univ_subs work, the updated data induarrays after der_univ_subs call gets written in the main memory, even though we don't need this at all.There are three ways we can fix this
du,dud, andd2uarrays into(SZ, n)sized temporary arrays. Then we pass temporary arrays into der_univ_subs, and at the end we use these temporaries to obtain finalrhs. This is the easiest solution but it may not be the best in terms of performance.duarrays as we do now, and pass a small temporary array as the output one. Becauseduarrays will be input arrays no data will be written in main memory. Then we can combine the temporaries to getrhs.du,dud, andd2u, and write the final resultrhs.