EtienneBachmann
commented
6 years ago Hi all,
Another idea to improve adjoint run speed is to merge the GPU kernels in the compute_kernels routine, where rho kernels and other kernels are separated. It should not affect the readability of the code. Also, it can be wise to introduce a flag COMPUTE_RHO_KERNELS, to avoid their calculation in case they are not needed. The associated computational cost is quite important in the case of acoustic (here outer core), because of the call to compute_gradient routines. To give an idea, on pure acoustic simulation, I obtain a 25% speedup on my purely acoustic adjoint simulation just by commenting the calculation of the rho acoustic kernel. I'm not specialist of the large runs on cluster, but I suspect that even in a perfectly balanced mesh, calibrated to run with more acoustic elements than elastic because of the compute forces routine, when it comes to kernels computation, the acoustic rho kernel calculation becomes an important bottleneck that slows down the whole simulation.
Best regards,
Etienne
komatits
danielpeter
From Etienne @EtienneBachmann , after fixing most of the GPU slowdown for very large adjoint runs already, here are some new ideas:
To further improve the GPU code, and maybe go below the 2:1 ratio on adjoint runs, here are few suggestions, that are based on my work in 2D :
In the update Newmark routine :
-The kernels update accel and update veloc (that uses accel for the update) can be merged, it will reduce the memory accesses.
Also, for the compute_forces_acoustic routine, it looks like with the NVIDIA P100 GPU there is a significant speedup merging the calls to direct and adjoint wavefields, around 40%. But if the acoustic elements only represent 5% of the computing time, I am not sure it is worth applying it.