search

SPECFEM / specfem2d

SPECFEM2D simulates forward and adjoint seismic wave propagation in two-dimensional acoustic, (an)elastic, poroelastic or coupled acoustic-(an)elastic-poroelastic media, with Convolution PML absorbing conditions.
GNU General Public License v3.0
200 stars 146 forks source link

Problem running in mpi mode #1134

Open gnpang opened 2 years ago

gnpang commented 2 years ago

Who may concern

I am running into a problem when I run the xspecfem2D using mpirun. I can run the code using 8 cores. However, if I would like to use more cores, say 16 cores, the xpecfem2D will run until the output says "Preparing array allocations arrays for elastic domains done initialization" Then it will hold it there, with no error output and no calculation results. I have talked to my department's IT group and we did not find any issues with the hardware side.

Does anyone of you meet this issue or can you help me figure it out?

danielpeter commented 2 years ago

could you attach your Par_file and output generated by the run?

gnpang commented 2 years ago

Please see the attachments for the Par_file I used and the output generated by the run.

On Thu, Nov 4, 2021 at 1:07 AM daniel peter @.***> wrote:

could you attach your Par_file and output generated by the run?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/geodynamics/specfem2d/issues/1134#issuecomment-960462618, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC2FQ4PCQMJBLMXOHVGA55TUKIIIZANCNFSM5HJVQNNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Guanning Pang, Ph. D. Postdoctoral Scholar Earth and Atmospheric Sciences Cornell University

Specfem 2-D Solver - MPI version

Running Git version of the code corresponding to commit 9e6c71277d9e563d85f100adce5ffb6388ca3383 dating From Date: Mon Sep 20 15:23:03 2021 +0300

There are 16 MPI processes Processes are numbered from 0 to 15

There is a total of 16 slices

NDIM = 2

NGLLX = 5 NGLLZ = 5

using single precision for the calculations

smallest and largest possible floating-point numbers are: 1.17549435E-38 3.40282347E+38

Reading the parameter file...

Title of the simulation: Test of SPECFEM2D with curved interfaces

Parameter file successfully read

Reading in SOURCE file...

Source 1 Position xs, zs = 50000.000000000000 800000.00000000000

Source type (1=force, 2=moment tensor): 1 Force source: Angle of the source (deg) = 0.0000000000000000

Time function type (1=Ricker, 2=First derivative, 3=Gaussian, 4=Dirac, 5=Heaviside, 6,7=ocean type, 8=Read from file, 9=burst, 10=Sinusoidal, 11=Ormsby): 1 Ricker wavelet (second-derivative): Frequency, delay = 1.0000000000000000 0.0000000000000000
Multiplying factor = 10000000000.000000

all sources are okay

Program SPECFEM2D:

Test of SPECFEM2D with curved interfaces

D a t e : 04 - 11 - 2021 T i m e : 08:37:18

C o n t r o l

 Number of spectral element control nodes. . .(npgeo) =   82369
 Number of space dimensions. . . . . . . . . . (NDIM) =       2

C o n t r o l

 Display frequency . . . .(NSTEP_BETWEEN_OUTPUT_INFO) =    100
 Color display . . . . . . . . . . . . . . . (colors) =      1
  == 0     black and white display              
  == 1     color display                        
 Numbered mesh . . . . . . . . . . . . . . .(numbers) =      0
  == 0     do not number the mesh               
  == 1     number the mesh

C o n t r o l

 Seismograms recording type . . . . . . .(seismotype) = 1
 Angle for first line of receivers. . . . .(anglerec) =   0.00

C o n t r o l

 Read external initial field. . . . . .(initialfield) =      F
 Add Bielak conditions (add_Bielak_conditions_bottom) =      F
 Add Bielak conditions .(add_Bielak_conditions_right) =      F
 Add Bielak conditions . .(add_Bielak_conditions_top) =      F
 Add Bielak conditions. .(add_Bielak_conditions_left) =      F
 Attenuation in solid . . .(ATTENUATION_VISCOELASTIC) =      T
 Attenuation in fluid . . (ATTENUATION_VISCOACOUSTIC) =      F
 Save grid in ASCII file or not . (output_grid_ASCII) =      F
 Save a file with total energy or not.(OUTPUT_ENERGY) =      F

C o n t r o l

 Vector display type . . . . . . . . .(imagetype_postscript) =      1
 Percentage of cut for vector plots. . . . . . . .(cutsnaps) =   1.00
 Subsampling of velocity model display. (subsamp_postscript) =      1

I t e r a t i o n s 

 Number of time iterations . . . . .(NSTEP) =   30000
 Time step increment. . . . . . . . . .(DT) =   4.000000E-03
 Total simulation duration . . . . . (ttot) =   1.200000E+02

reading mesh databases: reading nodal coordinates...

-- Spectral Elements --

 Number of spectral elements . . . . . . . . .  (nspec) = 640998
 Number of control nodes per element . . . . . (ngnod) =      4
 Number of points in X-direction . . . . . . . (NGLLX) =      5
 Number of points in Y-direction . . . . . . . (NGLLZ) =      5
 Number of points per element. . . . . . (NGLLX*NGLLZ) =     25
 Number of points for display . . . . . . (pointsdisp) =      6
 Number of element material sets . . . . . . . (numat) =      3
 Number of acoustic forcing elements (nelem_acforcing) =      0
 Number of acoustic free surf (nelem_acoustic_surface) =      0

reading attenuation setup... reading material properties..

M a t e r i a l s e t s : 2 D (p o r o) e l a s t i c i t y

 Number of material sets . . . . . . (numat) =     3

 -------------------------------
 -- Acoustic (fluid) material --
 -------------------------------
 Material set number. . . . . . . . (jmat) =     1
 P-wave velocity. . . . . . . . . . . (cp) = 1.02830000E+04
 Mass density. . . . . . . . . . (density) = 1.21240000E+04
 Bulk modulus Kappa . . . . . . . .(kappa) = 1.28199284E+12
 QKappa_attenuation. . . . . . . .(QKappa) = 9.99900000E+03
 Qmu_attenuation. . . . . . . . . . .(Qmu) = 9.99900000E+03

 ----------------------------------------------------
 -- Material described by external tomography file --
 ----------------------------------------------------
 Material set number. . . . . . . . (jmat) =     2

 ----------------------------------------
 -- Elastic (solid) isotropic material --
 ----------------------------------------
 Material set number. . . . . . . . (jmat) =     3
 P-wave velocity. . . . . . . . . . . (cp) = 1.10370000E+04
 S-wave velocity. . . . . . . . . . . (cs) = 3.50500000E+03
 Mass density. . . . . . . . . . (density) = 1.27040000E+04
 Poisson's ratio. . . . . . . . .(poisson) = 4.43919536E-01
 First Lame parameter Lambda. . . (lambda) = 1.23540453E+12
 Second Lame parameter Mu. . . . . . .(mu) = 1.56068958E+11
 Bulk modulus Kappa . . . . . . . .(kappa) = 1.39147349E+12
 Young's modulus E. . . . . . . . .(young) = 4.51332901E+11
 QKappa_attenuation .  . . . . . .(QKappa) = 3.00000000E+02
 Qmu_attenuation . . . . . . . . . . (Qmu) = 3.00000000E+02

reading interfaces informations... number of interfaces = 3 reading absorbing boundary... number of absorbing elements = 600

any absorbing boundary flag = T PML boundary flag = F Stacey boundary flag = T

Absorbing boundaries: using Stacey absorbing boundary conditions

Total number of absorbing elements: 3208 nspec_left = 1604 nspec_right = 0 nspec_bottom = 1604 nspec_top = 0

reading acoustic forcing... acoustic forcing = F number of acoustic forcing boundary elements = 0 reading free surface information... number of acoustic free surface boundary elements = 0 reading coupling surfaces... number of fluid-solid edges = 268 number of fluid-poro edges = 0 number of solid-poro edges = 0 reading tangential curves... number of tangential curve nodes = 0 reading axial elements... number of axial elements = 0

done reading mesh databases

Setting up mesh

Generating global mesh numbering (fast version)...

Mesh numbering: Total number of elements: 640998

Total number of acoustic elements = 160000 Total number of elastic/visco/poro elements = 480998

Approximate total number of grid points in the mesh (with a few duplicates coming from MPI buffers): 10285388

Approximate number of acoustic degrees of freedom in the mesh: 2567344 Approximate number of elastic degrees of freedom in the mesh: 15436089 (there are 2 degrees of freedom per point for elastic elements)

Approximate total number of degrees of freedom in the mesh (sum of the two values above): 18003433

(for simplicity viscoelastic or poroelastic elements, if any, are counted as elastic in the above three estimates; in reality they have more degrees of freedom)

Mesh dimensions: Xmin,Xmax of the whole mesh = -200000.00000000003 600000.00000000000
Zmin,Zmax of the whole mesh = 0.0000000000000000 800000.00000000012

Assigning an external velocity and density model model: tomo tomographic model file: ./DATA/tomo_file.xyz Number of grid points = NX*NZ: 1922801

done

setup material arrays setting up anisotropic arrays

Domains: total number of acoustic elements = 160000 total number of elastic elements = 480998 total number of poroelastic elements = 0

Mixed acoustic/elastic simulation

Beginning of fluid/solid edge detection Checking fluid/solid edge topology... End of fluid/solid edge detection

excluding common points between acoustic absorbing edges and acoustic/elastic matching interfaces, if any

Setting up MPI communication arrays

maximum number of MPI interfaces (for a single slice) = 7

total MPI interface points: 46048

checking acoustic interfaces: total number of global acoustic interface points: 4316 total number of global points assembled by acoustic MPI interfaces: 4316 interfaces okay

main process: number of MPI interfaces in acoustic domain = 1 number of MPI interfaces in elastic domain = 3 number of MPI interfaces in poroelastic domain = 0

number of outer elements = 657 number of inner elements = 81082

percentage of outer elements 0.803779602 % percentage of inner elements 99.1962204 %

number of global nodes in outer elements = 13145 number of global nodes in inner elements = 1299821

determining communication phases: elastic domains: total number of outer/inner elements = 10371 470627 total percentage of outer elements 2.15614319 % total percentage of inner elements 97.8438568 %

acoustic domains: total number of outer/inner elements = 1078 158922 total percentage of outer elements 0.673751831 % total percentage of inner elements 99.3262482 %

All mesh setup done successfully

sources: Total number of sources: 1

Setting parameters for source 1

 Source Type. . . . . . . . . . . . . . = Collocated Force
 X-position (meters). . . . . . . . . . =    5.0000000000E+04
 Y-position (meters). . . . . . . . . . =    8.0000000000E+05
 Fundamental frequency (Hz) . . . . . . =    1.0000000000E+00
 Time delay (s) . . . . . . . . . . . . =    0.0000000000E+00
 Multiplying factor . . . . . . . . . . =    1.0000000000E+10
 Angle from vertical direction (deg). . =    0.0000000000E+00

Source: 1 Onset time. . . . . . = -1.2000000000000000
Fundamental period. . = 1.0000000000000000
Fundamental frequency = 1.0000000000000000
The onset time is ok

Source: 1

locating force source

Force source: original x: 50000.0000
original z: 800000.000
Closest estimate found: 0.00000000 m away in rank 0 in element 10344 in acoustic domain at xi,gamma coordinates = -1.0000000000000000 -1.0000000000000000

end of force source detection

receivers:

Total number of receivers = 21

locating receivers

reading receiver information from the ./DATA/STATIONS file

Station # 1 AA S0001
original x: 445000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 185 in acoustic domain at xi,gamma coordinates = -1.0000000000000000 1.0000000000000000
Distance from source: 398151.969 m

Station # 2 AA S0002
original x: 445500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 185 in acoustic domain at xi,gamma coordinates = -1.0000000000000000 0.0000000000000000
Distance from source: 398648.031 m

Station # 3 AA S0003
original x: 446000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 185 in acoustic domain at xi,gamma coordinates = -1.0000000000000000 -1.0000000000000000
Distance from source: 399144.094 m

Station # 4 AA S0004
original x: 446500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 2464 in acoustic domain at xi,gamma coordinates = 0.0000000000000000 -1.0000000000000000
Distance from source: 399640.156 m

Station # 5 AA S0005
original x: 447000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 2464 in acoustic domain at xi,gamma coordinates = 1.0000000000000000 -1.0000000000000000
Distance from source: 400136.219 m

Station # 6 AA S0006
original x: 447500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 2465 in acoustic domain at xi,gamma coordinates = -1.0000000000000000 0.0000000000000000
Distance from source: 400632.312 m

Station # 7 AA S0007
original x: 448000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 2465 in acoustic domain at xi,gamma coordinates = -1.0000000000000000 -1.0000000000000000
Distance from source: 401128.406 m

Station # 8 AA S0008
original x: 448500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 28382 in acoustic domain at xi,gamma coordinates = 0.0000000000000000 -1.0000000000000000
Distance from source: 401624.500 m

Station # 9 AA S0009
original x: 449000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 28379 in acoustic domain at xi,gamma coordinates = -1.0000000000000000 1.0000000000000000
Distance from source: 402120.625 m

Station # 10 AA S0010
original x: 449500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 28379 in acoustic domain at xi,gamma coordinates = -1.0000000000000000 0.0000000000000000
Distance from source: 402616.750 m

Station # 11 AA S0011
original x: 450000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 5947 in acoustic domain at xi,gamma coordinates = -1.0000000000000000 -1.0000000000000000
Distance from source: 403112.875 m

Station # 12 AA S0012
original x: 450500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 5947 in acoustic domain at xi,gamma coordinates = -1.0000000000000000 0.0000000000000000
Distance from source: 403609.031 m

Station # 13 AA S0013
original x: 451000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 5946 in acoustic domain at xi,gamma coordinates = 1.0000000000000000 -1.0000000000000000
Distance from source: 404105.188 m

Station # 14 AA S0014
original x: 451500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 5946 in acoustic domain at xi,gamma coordinates = 0.0000000000000000 -1.0000000000000000
Distance from source: 404601.344 m

Station # 15 AA S0015
original x: 452000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 4199 in acoustic domain at xi,gamma coordinates = -1.0000000000000000 -1.0000000000000000
Distance from source: 405097.531 m

Station # 16 AA S0016
original x: 452500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 4199 in acoustic domain at xi,gamma coordinates = 0.0000000000000000 -1.0000000000000000
Distance from source: 405593.688 m

Station # 17 AA S0017
original x: 453000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 4196 in acoustic domain at xi,gamma coordinates = -1.0000000000000000 1.0000000000000000
Distance from source: 406089.906 m

Station # 18 AA S0018
original x: 453500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 4196 in acoustic domain at xi,gamma coordinates = -1.0000000000000000 0.0000000000000000
Distance from source: 406586.094 m

Station # 19 AA S0019
original x: 454000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 4196 in acoustic domain at xi,gamma coordinates = -1.0000000000000000 -1.0000000000000000
Distance from source: 407082.312 m

Station # 20 AA S0020
original x: 454500.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 15570 in acoustic domain at xi,gamma coordinates = 0.0000000000000000 -1.0000000000000000
Distance from source: 407578.531 m

Station # 21 AA S0021
original x: 455000.000
original z: 750000.000
Closest estimate found: 0.00000000 m away in rank 13 in element 15567 in acoustic domain at xi,gamma coordinates = -1.0000000000000000 1.0000000000000000
Distance from source: 408074.750 m

end of receiver detection

found a total of 21 receivers, this is okay

Preparing timerun:

Preparing timerun constants

Preparing array allocations arrays for acoustic domains arrays for elastic domains done initialization

danielpeter commented 2 years ago

thanks for the outputs! it seems to be a problem with the Stacey absorbing boundary setup becoming blocking when there are partitions without boundary elements.

please try the updated version here on the devel branch - it should work now with PR #1136

gnpang commented 2 years ago

Thank you for the updates. I tried in on our nodes with 16 cores. It is working fine now.

On Wed, Nov 10, 2021 at 12:10 AM daniel peter @.***> wrote:

thanks for the outputs! it seems to be a problem with the Stacey absorbing boundary setup becoming blocking when there are partitions without boundary elements.

please try the updated version here on the devel branch - it should work now with PR #1136 https://github.com/geodynamics/specfem2d/pull/1136

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/geodynamics/specfem2d/issues/1134#issuecomment-964793283, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC2FQ4KUL7BKVGMGQZKQEBTULH5DHANCNFSM5HJVQNNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Guanning Pang, Ph. D. Postdoctoral Scholar Earth and Atmospheric Sciences Cornell University