search

Exawind / amr-wind

AMReX-based structured wind solver
https://exawind.github.io/amr-wind
Other
106 stars 82 forks source link

Boundary plane input on GPUs #396

Closed shashankNREL closed 3 years ago

shashankNREL commented 3 years ago

I am trying to run the abl_bndry_input regression test case on GPUs (Eagle and Summit). I first ran the abl_bndry_output test case to generate all the relevant files to run the abl_bndry_input regression test case. The case works without any problem on CPUs on Eagle. The same case with GPU build on eagle (and Summit) gives the wrong initial residual for Mac projection and crashes in the MAC_projection step. This issue can be reproduced using the abl_bndry_input regression test on GPU builds. @marchdf : Can you verify that this case worked on GPUs in the past. Is that something that you could look into? @jrood-nrel : Can we set up a suite of regression case testing using GPUs on Eagle.

jrood-nrel commented 3 years ago

We test nightly on Eagle's GPUs.

marchdf commented 3 years ago

Hey Shashank, I believe I tested this with Eagle when I initially did this. But clearly it is not anymore... I will look into it.

jrood-nrel commented 3 years ago

@michaeljbrazell I see abl_bndry_input as a test directory but it's not in the actual list.

sayerhs commented 3 years ago

@jrood-nrel abl_bndry_{input,output} requires AMR-Wind built with NetCDF support. So these tests are skipped in the GPU builds.

PaulMullowney commented 3 years ago

Regarding mac_proj and with hypre as a bottom solver (on GPUs), I run abl_godunov with:

mac_proj.verbose = 1 mac_proj.mg_verbose = 1 mac_proj.mg_cg_verbose = 1 mac_proj.mg_rtol = 1.0e-8 mac_proj.mg_atol = 1.0e-13

mac_proj.max_coarsening_level = 0 mac_proj.bottom_solver = hypre

mac_proj.bottom_verbose = 0 mac_proj.bottom_rtol = 1.0e-8 mac_proj.bottom_atol = 1.0e-16 mac_proj.bottom_maxiter = 50 mac_proj.hypre_namespace = "mac_proj.hypre" mac_proj.hypre.hypre_solver = GMRES mac_proj.hypre.hypre_preconditioner = BoomerAMG mac_proj.hypre.verbose = 2 mac_proj.hypre.bamg_verbose = 1 mac_proj.hypre.num_krylov = 40 mac_proj.hypre.bamg_coarsen_type = 8 mac_proj.hypre.bamg_interp_type = 6 mac_proj.hypre.bamg_relax_type = 11 mac_proj.hypre.bamg_num_sweeps = 1 mac_proj.hypre.bamg_max_levels = 10 mac_proj.hypre.bamg_cycle_type = 1 mac_proj.hypre.bamg_relax_order = 0 mac_proj.hypre.bamg_trunc_factor = 0.1 mac_proj.hypre.bamg_strong_threshold = 0.4 mac_proj.hypre.bamg_agg_num_levels = 2 mac_proj.hypre.bamg_agg_interp_type = 7 mac_proj.hypre.bamg_agg_pmax_elmts = 3 mac_proj.hypre.bamg_pmax_elmts = 3 mac_proj.hypre.bamg_keep_transpose = 1 mac_proj.hypre.recompute_preconditioner = 0 mac_proj.hypre.write_matrix_files = 0

sayerhs commented 3 years ago

@PaulMullowney this test is not failing due to hypre. It is using native AMReX solvers. This particular set has a different inflow condition that is read from a NetCDF file, and we suspect that there is some issue with setting the BC itself. As I mentioned in #391, if you can update the ABL regression test with hypre solvers with these input settings, that would be great.