feature: parallel solve subspace diagonalization in dav_subspace

pxlxingliang commented 1 day ago

Add function Diago_HS_para() to diago the H/S by genelpa or scalapack. The function will perform 2D block distribution of data, call elpa/scalapack for solution, and collect eigenvectors to rank0.
Add a unified function interface pxxxgvx_diag() for calling scalapack to solve the generalized eigenvalue problem with double/complex/float/complex.
Add a new INPUT parameter diag_subspace_method, which support to set the subspace diagonalization method in dav_subspace to use elpa or scalapack to solve.

Reminder

[ ] Have you linked an issue with this pull request?
[ ] Have you added adequate unit tests and/or case tests for your pull request?
[ ] Have you noticed possible changes of behavior below or in the linked issue?
[ ] Have you explained the changes of codes in core modules of ESolver, HSolver, ElecState, Hamilt, Operator or Psi? (ignore if not applicable)

Linked Issue

Fix #5480

Unit Tests and/or Case Tests for my changes

A unit test is added for each new feature or bug fix.

What's changed?

Example: My changes might affect the performance of the application under certain conditions, and I have tested the impact on various scenarios...

Any changes of core modules? (ignore if not applicable)

Example: I have added a new virtual function in the esolver base class in order to ...

haozhihan commented 12 hours ago

Diago_HS_para implements parallel solution for the subspace process in dav-subspace. If this is efficient, can we just fix one solution method to reduce the burden of users?

haozhihan commented 11 hours ago

I have a brief insight regarding this PR:

This process involves a transformation of parallel strategy.

from basis parallelism to 2D block parallelism
from 2D block parallelism to basis parallelism

Can this transformation of parallel strategy be more general?

If the H and S matrices of LCAO are solved by iterative method (like cg, dav, and so on, usually used for plane wave basis), it will also involve almost the same transformation of parallel strategy.

pxlxingliang commented 11 hours ago

Diago_HS_para implements parallel solution for the subspace process in dav-subspace. If this is efficient, can we just fix one solution method to reduce the burden of users?

The parallel diagonization is not always more efficient, it is related to the system size, parallel cores, and efficiency of parallel communication, etc. I have done some tests in #5480.

pxlxingliang commented 11 hours ago

I have a brief insight regarding this PR:

This process involves a transformation of parallel strategy.

from basis parallelism to 2D block parallelism

from 2D block parallelism to basis parallelism

Can this transformation of parallel strategy be more general?

If the H and S matrices of LCAO are solved by iterative method (like cg, dav, and so on, usually used for plane wave basis), it will also involve almost the same transformation of parallel strategy.

The transformation of different 2D block distribution can be realized easily by call the scalapack function Cpigemr2d() (a uniform interface for different data type is here: https://github.com/deepmodeling/abacus-develop/blob/develop/source/module_base/scalapack_connector.h#L158). While the transformation for basis parallelism is strongly related to the self-defined class in ABACUS (like psi?), may be a class function of psi to do the transformation is better.

deepmodeling / abacus-develop