search

hypre-space / hypre

Parallel solvers for sparse linear systems featuring multigrid methods.
https://www.llnl.gov/casc/hypre/
Other
700 stars 193 forks source link

Hypre, Umpire and hipMemGetInfo #777

Open PaulMullowney opened 2 years ago

PaulMullowney commented 2 years ago

Hi All, I'm using Hypre with Umpire enabled for AMD GPUs (rocm 5.1.0). In my application, I query the amount of free memory available with hipMemGetInfo after key steps (matrix creation, vector creation, preconditioner setup, ...). The results of these calls when not using Umpire are:

Before BuildMatrix : rank 0 of 8, free memory=63.98 GB, total memory=63.98 GB Before hypre_CSRMatrixGpuRawSpMVAnalysis : rank 0 of 8, free memory=55.32 GB, total memory=63.98 GB Before RHS IJVectorCreate : rank 0 of 8, free memory=54.90 GB, total memory=63.98 GB After Preconditioner Setup : rank 0 of 8, free memory=43.68 GB, total memory=63.98 GB

This seems reasonable to me given the size of the problem. Once I build with Hypre with Umpire and a 4GB Device Memory Pool, I see:

Before BuildMatrix : rank 0 of 8, free memory=63.98 GB, total memory=63.98 GB Before hypre_CSRMatrixGpuRawSpMVAnalysis : rank 0 of 8, free memory=.8517 GB, total memory=63.98 GB Before RHS IJVectorCreate : rank 0 of 8, free memory=.8517 GB, total memory=63.98 GB After Preconditioner Setup : rank 0 of 8, free memory=17.73 GB, total memory=63.98 GB

My question is, should I expect hipMemGetInfo to work properly when using device memory pools? Thanks -Paul

PaulMullowney commented 1 year ago

Anyone have any insight on this?

As a follow up, should I expect device memory pools larger than 4GB to work?