search

SPECFEM / specfem3d

SPECFEM3D_Cartesian simulates acoustic (fluid), elastic (solid), coupled acoustic/elastic, poroelastic or seismic wave propagation in any type of conforming mesh of hexahedra (structured or not).
GNU General Public License v3.0
389 stars 223 forks source link

#1344 CUDA memory error! #1679

Closed brioglade closed 4 months ago

brioglade commented 4 months ago

hi

https://github.com/SPECFEM/specfem3d/issues/1344

~/specfem3d/EXAMPLES/applications/fault_examples/tpv15

This example can't enable GPU true. I use another computer with a relatively large memory, but still encounter the following error. Although the above example used 32 $NPROCs, it still took about 5 hours.

CUDA error !!!!! !!!!! at CUDA call error code: # 2402 Abort(1) on node 1 (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 Abort(1) on node 3 (rank 3 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3 Abort(1) on node 4 (rank 4 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 4 Abort(1) on node 5 (rank 5 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 5 Abort(1) on node 6 (rank 6 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 6

How to fix it? Thank you very much.

danielpeter commented 4 months ago

you need more GPUs... the simulation is too big for your card.