search

SPECFEM / specfem3d

SPECFEM3D_Cartesian simulates acoustic (fluid), elastic (solid), coupled acoustic/elastic, poroelastic or seismic wave propagation in any type of conforming mesh of hexahedra (structured or not).
GNU General Public License v3.0
398 stars 225 forks source link

use PT-Scotch (the parallel version of SCOTCH) in SPECFEM3D instead of (or as an option in addition to) the serial version of SCOTCH #19

Closed mpbl closed 10 years ago

mpbl commented 10 years ago

use PT-Scotch (the parallel version of SCOTCH) in SPECFEM3D instead of (or as an option in addition to) the serial version of SCOTCH; because decomposing very large 3D meshes on a serial machine (on a single processor core) can be a severe limitation and for some meshes that will be a major bottleneck.

Support ParMetis as well because there is a new, improved version that has been released by Karypis et al recently. To do that, Jeroen sent a Fortran bug fix (a patch to use to call METIS from Fortran) to Matthieu by email on Jan 16, 2013.

What would need to be done is switch to PT-Scotch version 6.0 and/or to ParMETIS (supporting both would be easy, since the subroutine calls are very similar; in the current serial code we support both Scotch and METIS).

To do that, one would need to rewrite "decompose_mesh.F90" in directory src/decompose_mesh. That is the only file that would need to change (in addition to part_decompose_mesh.f90, which is called by decompose_mesh.F90). Unfortunately that file is not very well written nor very well commented. Thus unfortunately that set of routines is not so easy to understand and modify. A bunch of things are hardwired.

Thus, this is not so easy to do (Daniel and I checked about a year ago). It would maybe be easier to rewrite 'decompose_mesh_scotch' from scratch as a new, MPI program instead of trying to add MPI support to the current and not so well written serial version. That program is relatively small, and what it does is relatively straightforward.

komatits commented 10 years ago

Duplicate of https://github.com/geodynamics/specfem3d/issues/129