Open zLi90 opened 2 years ago
@jennloe
Actually a good start would be to re-run this with finer timers, for instance using Kokkos-tools simple-kernel-timer. You just need to build the tool and then set: export KOKKOS_PROFILE_LIBRARY=${HOME}/kokkos-tools/profiling/simple-kernel-timer/kp_kernel_timer.so
I will start looking at this...
@zLi90 What size problem were you running on to get these timings? How sparse is the matrix? What was the convergence tolerance? How many iterations were you running? And what size of problem do you ultimately want to be able to run?
@jennloe We are running a 3D transient subsurface flow model. This test case has 40x24x32=30720 grid cells, resulting a 7-diagonal matrix. The tolerance is 1e-8 and it takes (more or less) 18 pcg iterations for each time step (we ran a total of 900 seconds with a time step dt=60s). We hope to be able to run large scale hydrological simulations that contain 1 million grid cells or more. Thanks!
Hi @jennloe , just wondering if you have figured out something? Do you have any suggestions on what we could try next? Thanks!
I'm sorry; I have not had more time to look at this, and I will not have time in the next few weeks. Thank you for your patience. @brian-kelley would you be able to look into this scaling?
I like Luc's comment above. @zLi90 can you enable the finer timers please?
@srajama1 @lucbv Thanks for the suggestion! I have tried the simple-kernel-timer but I am not sure how to decode the output file (or perhaps I did something wrong?)
For example, I got an output file with the first few lines like this:
We are using the provided pcg solver in kokkoskernels for our hydrological model. We found that when we activate openmp, the scaling does not look good.
We simply call
run_pcg<Kokkos::OpenMP>(domsub.nCellDomain, statesub, domsub, timers);
and inrun_pcg
, we call:KokkosKernels::Experimental::Example::pcgsolve(kh, matA, vecB, vecX, cg_iteration_limit, cg_iteration_tolerance, & cg_result, true, clusterSize, useSequential);
Kokkos::fence();
The
pcgsolve
takes about 90% of the computational time and it doesn't scale well when we add threads. But since it is provided by kokkoskernels, it is difficult for us to find out what might be the reason. We were wondering if there are additional steps we are missing that can enhance scaling performance of the pcg solver?We also want to ask if you have any recommendations on choosing an linear system solver (perhaps also a nonlinear solver) that is compatible with kokkos and has good scaling performance when using openmp and mpi?
Here I attached a figure that shows the speedup of our model (the blue dots), using the![Screen Shot 2021-10-26 at 8 42 24 PM](https://user-images.githubusercontent.com/18181974/138883947-43eb652e-f597-4dc3-90cd-a4940c186102.png)
pcgsolve
. Thank you so much in advance!