search

xiaoyeli / superlu_dist

Distributed memory, MPI based SuperLU
https://portal.nersc.gov/project/sparse/superlu/
Other
185 stars 65 forks source link

Error while redistributing A on the 2D process mesh #26

Open aithalab opened 6 years ago

aithalab commented 6 years ago

Hi,

I have been using superlu_dist to solve a complex-valued block-tridiagonal linear system that arises from discretizing a 2D variable coefficient Poisson equation using 2nd order central difference scheme for a few months. The coefficient matrix, A, is symmetric and positive definite. The linear system is complex valued because a 3D Poisson equation has been reduced to 2D by performing Fourier transform. I have also installed METIS and ParMETIS. I have tested other direct solvers for sparse matrices such as MUMPS and PaStiX (which are developed for symmteric matrices) and found that superlu_dist outperformed them both in terms of computational speed and memory footprint.

The library performs without any issues when the dimension of A is O(1e6). This is the case when the 3D grid consists of (256,128,64) grid points. However, when I increase the grid size to (512,256,128) grid points, I get the following error message: Calloc fails for SPA dense[]. at line 761 in file /libraries/SuperLU_DIST_5.1.3/SRC/pzdistribute.c

Do you have any suggestions to fix this? Also, since the coefficient matrix, A, is symmetric and positive definite I expect U to be the transpose of L and thus there is no need to store them both. Is there a way to incorporate this in the solver?

Thanks

aithalab commented 6 years ago

Tagging contributors @xiaoyeli @gchavez2 @jrobcary

xiaoyeli commented 6 years ago

How many nodes / processes are you using?

You can try one of the two things: 1) Use parallel symbolic factorization option, see FAQ http://crd-legacy.lbl.gov/~xiaoye/SuperLU/faq.html#superlu_dist:parsymbfact 2) Reduce the panel size, i.e., the #3 parameter in SRC/sp_ienv.c(), which can be reset by env. variable "NSUP". (The default is 128)

Sherry Li

On Fri, Jan 26, 2018 at 9:48 AM, aithalab notifications@github.com wrote:

Hi,

I have been using superlu_dist to solve a complex-valued block-tridiagonal linear system that arises from discretizing a 2D variable coefficient Poisson equation using 2nd order central difference scheme for a few months. The coefficient matrix, A, is symmetric and positive definite. The linear system is complex valued because a 3D Poisson equation has been reduced to 2D by performing Fourier transform. I have also installed METIS and ParMETIS. I have tested other direct solvers for sparse matrices such as MUMPS and PaStiX (which are developed for symmteric matrices) and found that superlu_dist outperformed them both in terms of computational speed and memory footprint.

The library performs without any issues when the dimension of A is O(1e6). This is the case when the 3D grid consists of (256,128,64) grid points. However, when I increase the grid size to (512,256,128) grid points, I get the following error message: Calloc fails for SPA dense[]. at line 761 in file /libraries/SuperLU_DIST_5.1.3/SRC/pzdistribute.c

Do you have any suggestions to fix this? Also, since the coefficient matrix, A, is symmetric and positive definite I expect U to be the transpose of L and thus there is no need to store them both. Is there a way to incorporate this in the solver?

Thanks

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/xiaoyeli/superlu_dist/issues/26, or mute the thread https://github.com/notifications/unsubscribe-auth/ALMq9wxUsOQ5I9MbbUiNJ-fI4i9UkYLLks5tOg_ngaJpZM4RupZm .