How to build and run PeleC using GPU?

[EarlFan]

Dear all,

Hi!

I want to build and run PeleC using GPU, however, I am not able to find any tutorial on installations relevant to GPU or the CUDA enviroment. Can anyone provide some tutorial? Any help will be appreciated!

Thanks!

Regards, Fan E

[baperry2]

It's hard to provide detailed instructions for GPU use as the details can vary from system to system. But if you want to run on a system with Nvidia GPUs using cuda and your system is set up properly, all you should need to do is compile as normal, but with USE_CUDA = TRUE (and USE_MPI = TRUE assuming you also want MPI support) in your GNUmakefile. I'd recommend trying this for the PMF case using the pmf-lidryer-cvode.inp input file.

When running on GPUs, certain simulation input parameters may benefit from being re-optimized for performance. In particular, you may want larger values for amr.blocking_factor and amr.max_grid_size, and you may want to look at different options for cvode.solve_type. Every problem is different so it's usually good to do a little experimentation.

[jrood-nrel]

It's useful to know that certain things need to be done for certain sites, and AMReX has some supported sites here https://github.com/AMReX-Codes/amrex/tree/development/Tools/GNUMake/sites . The machine query logic is here https://github.com/AMReX-Codes/amrex/blob/development/Tools/GNUMake/Make.machines .

[EarlFan]

Dear all,

Thank you for your assistance!

I try to compile PeleC with nvcc on the WSL but have encountered some challenges, particularly with the Sundial package. Currently, I am able to run PeleC on CPUs without issues, but I am eager to explore the capabilities of GPU acceleration.

If it is OK, I would like to keep this issue open to share my future experiences regarding the use of PeleC with GPU computing.

Regards, Fan E

[baperry2]

Yeah that's fine to leave this issue open and add more detail on any issues you have running on GPUs, which we can then try to address.

[SRkumar97]

It's hard to provide detailed instructions for GPU use as the details can vary from system to system. But if you want to run on a system with Nvidia GPUs using cuda and your system is set up properly, all you should need to do is compile as normal, but with USE_CUDA = TRUE (and USE_MPI = TRUE assuming you also want MPI support) in your GNUmakefile. I'd recommend trying this for the PMF case using the pmf-lidryer-cvode.inp input file.

When running on GPUs, certain simulation input parameters may benefit from being re-optimized for performance. In particular, you may want larger values for amr.blocking_factor and amr.max_grid_size, and you may want to look at different options for cvode.solve_type. Every problem is different so it's usually good to do a little experimentation.

Hello! I have a doubt to clarify. When I first tested the code in CPU parallel mode, by running the basic PMF testcase I had not set MPI=TRUE in the example.inp file. But still the mpirun -np command worked out, to run the PeleC executable. Did I miss out anything?

[jrood-nrel]

mpirun will run any application with multiple instances, for example try mpirun -np 8 echo "hello".

Without MPI enabled in PeleC it will run the same application with np instances but they won't communicate to solve a single problem. Without MPI enabled mpirun will run the same problem in multiple instances with no benefit of concurrency.

[baperry2]

Note that when you compile for MPI, you should have USE_MPI = TRUE in your GNUmakefile, and MPI should appear in the name of the PeleC executable that gets generated. No changes are needed in the input files to run with MPI. But if the executable doesn't have MPI in the name, you generated a serial executable and it will run independent instances as mentioned by @jrood-nrel.

[SRkumar97]

Thanks for your clarifications on this! @jrood-nrel @baperry2 .