search

SPECFEM / specfem3d_globe

SPECFEM3D_GLOBE simulates global and regional (continental-scale) seismic wave propagation for Earth, Mars & Moon.
https://specfem.org
GNU General Public License v3.0
91 stars 96 forks source link

🐛 [BUG] - <Results of forward modelling for model 1D_transversely_isotropic_prem_onecrust don't match between branches Main and Devel> #846

Closed dosoe closed 1 month ago

dosoe commented 1 month ago

Description

I have run specfem3d_globe for the main and the devel branches for an event and a handful of stations and the results are not matching between the two runs. Here is a plot of the output, comparing the results for the mentioned event and stations (see reproduction steps) with the outputs filtered with the same filter, a bandpass with corner periods 500, 400, 50 and 40 seconds. Black is specfem3d_globe 8.0, red is 8.1 (devel branch, both were cloned today 16/09/2024) and blue is the difference. Both red and black curves have been normalised by the maximum of the black curve. As can be seen, there is a significant discrepancy between the two.
B012796B_section_Z_synth_new_1D_prem_main_vs_devel

I can provide the plotting code it requested, as well as the full logs.

This is problematic for me because I am attempting to include another model into specfem and I need to benchmark it. I have been told to use the devel branch, but if the outputs of devel are incorrect, it is impossible for me to benchmark the results.

Affected SPECFEM3D_GLOBE version

8.0 and 8.1

Your software and hardware environment

Run on cluster Anvil on 3 nodes (384 cores), compilers openmpi/4.0.6, intel/19.0.5.281

Reproduction steps

1. Go to 'DATA/Par_file'
2. Change settings 'NCHUNKS                         = 6
# number of elements at the surface along the two sides of the first chunk
# (must be multiple of 16 and 8 * multiple of NPROC below)
NEX_XI                          = 112
NEX_ETA                         = 112

# number of MPI processors along the two sides of the first chunk
NPROC_XI                        = 7
NPROC_ETA                       = 7
MODEL                           = 1D_transversely_isotropic_prem_onecrust

# parameters describing the Earth model
OCEANS                          = .false.
ELLIPTICITY                     = .false.
TOPOGRAPHY                      = .false.
GRAVITY                         = .false.
ROTATION                        = .false.
ATTENUATION                     = .false.
# save mesh files to check the mesh
SAVE_MESH_FILES                 = .true.

ABSORBING_CONDITIONS            = .false.

'
3. Use data files 'STATIONS:
___G._CAN       US         -35.3187141     148.9963226  0.00000000      0.00000000
___G._DRV       US         -66.6649094     140.0020752  0.00000000      0.00000000
___G._INU       US          35.3499985     137.0290070  0.00000000      0.00000000
___G.NOUC       US         -22.0986309     166.3066559  0.00000000      0.00000000
___G._PPT       US         -17.5690002    -149.5760040  0.00000000      0.00000000
___G._PVC       US         -17.7399998     168.3119965  0.00000000      0.00000000
__GE._PMG       US          -9.4091997     147.1539001  0.00000000      0.00000000
__IC._BJT       US          40.0182991     116.1679001  0.00000000      0.00000000
__IC._XAN       US          34.0312996     108.9236984  0.00000000      0.00000000
__II.MSVF       US         -17.7448006     178.0527954  0.00000000      0.00000000
__II._TAU       US         -42.9099007     147.3204041  0.00000000      0.00000000
__II.WRAB       US         -19.9335995     134.3600006  0.00000000      0.00000000
__IU._AFI       US         -13.9084997    -171.7826996  0.00000000      0.00000000
__IU.CHTO       US          18.8141003      98.9442978  0.00000000      0.00000000
__IU._DAV       US           7.0696998     125.5791016  0.00000000      0.00000000
__IU._HNR       US          -9.4386997     159.9474945  0.00000000      0.00000000
__IU.MAJO       US          36.5456696     138.2040558  0.00000000      0.00000000
__IU.NWAO       US         -32.9277000     117.2389984  0.00000000      0.00000000
__IU._PMG       US          -9.4047003     147.1596985  0.00000000      0.00000000
__IU._RAR       US         -21.2124996    -159.7733002  0.00000000      0.00000000
__IU.SNZO       US         -41.3087006     174.7042999  0.00000000      0.00000000
__IU.TATO       US          24.9734993     121.4971008  0.00000000      0.00000000
__PS._OGS       US          27.0569992     142.2030029  0.00000000      0.00000000
__PS.PATS       US           6.8367000     158.3152008  0.00000000      0.00000000
__PS._TSK       US          36.2108002     140.1096954  0.00000000      0.00000000
CMTSOLUTION:
PDEW 1996 1 27 17 48 10, 9.06 126.81 19.0 5.95 5.95 EVENT NAME
event name: B012796B
time shift: 0
half duration: 2.2
latitude: 9.06
longitude: 126.81
depth: 19.0
Mrr: 7.98e+24
Mtt: -4.08e+23
Mpp: -7.57e+24
Mrt: 5.48e+23
Mrp: 7.15e+24
Mtp: -2.11e+23
'
4. See error

Screenshots

![B012796B_section_Z_synth_new_1D_prem_main_vs_devel](https://github.com/user-attachments/assets/16d8f529-cd37-4d3d-a53a-61a469abca45)

Logs

output_mesher.txt for main branch: 

 ****************************
 *** Specfem3D MPI Mesher ***
 ****************************

 Version: v8.1.0-1-gb6fa4b6b

 Planet: Earth

 There are          294  MPI processes
 Processes are numbered from 0 to          293

 There are          112  elements along xi in each chunk
 There are          112  elements along eta in each chunk

 There are            7  slices along xi in each chunk
 There are            7  slices along eta in each chunk
 There is a total of           49  slices in each chunk
 There are            6  chunks in the global mesh
 There is a total of          294  slices in the global mesh

 NGLLX =            5
 NGLLY =            5
 NGLLZ =            5

 Shape functions defined by NGNOD =           27  control nodes
 Surface shape functions defined by NGNOD2D =            9  control nodes

 model: 1D_transversely_isotropic_prem_onecrust
   no oceans
   no ellipticity
   no surface topography
   no self-gravitation
   no rotation
   no attenuation

   no 3-D lateral variations in the mantle
   no heterogeneities in the mantle
   no crustal variations
   using one layer only in crust
   incorporating transverse isotropy
   no inner-core anisotropy
   no general mantle anisotropy

 Reference radius of the globe used is    6371.00000000000       km

 Central cube is at a radius of    950.000000000000       km
 creating global slice addressing

 Spatial distribution of the slices
                       48   41   34   27   20   13    6
 ...
                      189  182  175  168  161  154  147

 additional mesh optimizations

 moho:
   no element stretching for 3-D moho surface

 internal topography 410/660:
   no element stretching for 3-D internal surfaces

 Radial Meshing parameters:
   NCHUNKS                =            6

   CENTER LAT/LON:            0.0000000E+00 /  0.0000000E+00
   GAMMA_ROTATION_AZIMUTH:    0.0000000E+00

   CHUNK WIDTH XI/ETA:         90.00000     /   90.00000    
   NEX XI/ETA:                       112 /         112

   NER_CRUST:                          1
   NER_80_MOHO:                        1
   NER_220_80:                         2
   NER_400_220:                        2
   NER_600_400:                        2
   NER_670_600:                        1
   NER_771_670:                        1
   NER_TOPDDOUBLEPRIME_771:           15
   NER_CMB_TOPDDOUBLEPRIME:            1
   NER_OUTER_CORE:                    16
   NER_TOP_CENTRAL_CUBE_ICB:           2
   SUPPRESS_CRUSTAL_MESH:    F

   R_CENTRAL_CUBE =    950.0000      km

 Mesh resolution:
   DT =   0.235000000000000     
   Minimum period =    39.71247      (s)

 *******************************************
 creating mesh in region            1
 this region is the crust and mantle
 *******************************************

 first pass

   ...allocating arrays 

   ...setting up layers 

   ...creating mesh elements 
   creating layer            1 out of           10
     number of regular elements  =            0
     number of doubling elements =          448
   setting tiso flags in mantle model
     10.0%    current clock (NOT elapsed) time is: 17h 24min 52sec
 ...
   creating layer           10 out of           10
     number of regular elements  =          256
    100.0%    current clock (NOT elapsed) time is: 17h 24min 52sec
   layers done

   number of elements (per slice)        =         1904
   total number of elements (all slices) =       559776

   ...creating global addressing
     total number of points            :       238000
     array memory required per process :    5.44738769531250      MB
     getting global points             : npointot =       238000  nspec = 
        1904
     creating indirect addressing
     ibool ok

   ...creating MPI buffers

 second pass

   ...allocating arrays 

   ...setting up layers 

   ...creating mesh elements 
   creating layer            1 out of           10
     number of regular elements  =            0
     number of doubling elements =          448
     10.0%    current clock (NOT elapsed) time is: 17h 24min 52sec
 ...
   creating layer           10 out of           10
     number of regular elements  =          256
    100.0%    current clock (NOT elapsed) time is: 17h 24min 52sec
   layers done

   number of elements (per slice)        =         1904
   total number of elements (all slices) =       559776

   ...fills global mesh points 

   ...checking mesh resolution and time step

 ----------------------------------
   Verification of mesh parameters:
 ----------------------------------
   Region is crust/mantle

   Min Vs =    3.200000      (km/s)
   Max Vp =    13.71662      (km/s)

   Max element edge size =    255.1778      (km)
   Min element edge size =    18.53292      (km)
   Max/min ratio =    13.76890    

   Max Jacobian eigenvalue ratio =   0.9997538    
   Min Jacobian eigenvalue ratio =   7.1573265E-02

   Minimum period resolved =    26.30711      (s)
   Minimum period resolved (empirical) =    39.71247      (s)
   Maximum suggested time step =   0.2100000      (s)

   for DT :   0.2350000      (s)
   Max stability for wave velocities =   0.6014503    
 ----------------------------------

   ...precomputing Jacobian

   ...creating chunk buffers

 ----- creating chunk buffers -----

 There are            7  slices along xi in each chunk
 There are            7  slices along eta in each chunk
 There is a total of           49  slices in each chunk
 There are            6  chunks
 There is a total of          294  slices in all the chunks

 There is a total of           84  messages to assemble faces between chunks

 Generating message            1  for faces out of           84
 ...
 Generating message           84  for faces out of           84

 all the messages for chunk faces have the right size

 Generating message            1  for corners out of            8
 ...
 Generating message            8  for corners out of            8

   ...preparing MPI interfaces

 crust/mantle region:
   #max of points in MPI buffers along xi npoin2D_xi =         3409
   #max of array elements transferred npoin2D_xi*NDIM =        10227

   #max of points in MPI buffers along eta npoin2D_eta =         3409
   #max of array elements transferred npoin2D_eta*NDIM =        10227

 crust mantle MPI:
   maximum interfaces:           8
   MPI addressing maximum interfaces:           8
   MPI addressing : all interfaces okay

   total MPI interface points :      4129944
   unique MPI interface points:      3885504
   maximum valence            :            3
   total unique MPI interface points:     3885504

   ...element inner/outer separation 

 for overlapping of communications with calculations:

 percentage of edge elements in crust/mantle    40.33614     %
 percentage of volume elements in crust/mantle    59.66386     %

   ...element mesh coloring 
   mesh coloring:  F

   ...creating mass matrix

   ...saving binary files

 calculated region volume:    3.506132    
                 top area:    12.56637    
             exact area:    12.5663706143592     
              bottom area:    3.749325    
             exact area:    3.74932546676467     

 *******************************************
 creating mesh in region            2
 this region is the outer core
 *******************************************

 first pass

   ...allocating arrays 

   ...setting up layers 

   ...creating mesh elements 
   creating layer            1 out of            2
     number of regular elements  =           28
     number of doubling elements =           32
     50.0%    current clock (NOT elapsed) time is: 17h 24min 53sec
   creating layer            2 out of            2
     number of regular elements  =          112
    100.0%    current clock (NOT elapsed) time is: 17h 24min 53sec
   layers done

   number of elements (per slice)        =          172
   total number of elements (all slices) =        50568

   ...creating global addressing
     total number of points            :        21500
     array memory required per process :   0.492095947265625      MB
     getting global points             : npointot =        21500  nspec = 
         172
     creating indirect addressing
     ibool ok

   ...creating MPI buffers

 second pass

   ...allocating arrays 

   ...setting up layers 

   ...creating mesh elements 
   creating layer            1 out of            2
     number of regular elements  =           28
     number of doubling elements =           32
     50.0%    current clock (NOT elapsed) time is: 17h 24min 53sec
   creating layer            2 out of            2
     number of regular elements  =          112
    100.0%    current clock (NOT elapsed) time is: 17h 24min 53sec
   layers done

   number of elements (per slice)        =          172
   total number of elements (all slices) =        50568

   ...fills global mesh points 

   ...checking mesh resolution and time step

 ----------------------------------
   Verification of mesh parameters:
 ----------------------------------
   Region is outer core

   Min Vp =    8.064794      (km/s)
   Max Vp =    10.35571      (km/s)

   Max element edge size =    263.6152      (km)
   Min element edge size =    70.57796      (km)
   Max/min ratio =    3.735093    

   Max Jacobian eigenvalue ratio =   0.9999157    
   Min Jacobian eigenvalue ratio =   0.1590487    

   Minimum period resolved =    21.87847      (s)
   Minimum period resolved (empirical) =    39.71247      (s)
   Maximum suggested time step =   0.6900000      (s)

   for DT :   0.2350000      (s)
   Max stability for wave velocities =   0.1862786    
 ----------------------------------

   ...precomputing Jacobian

   ...creating chunk buffers

 ----- creating chunk buffers -----

 There are            7  slices along xi in each chunk
 There are            7  slices along eta in each chunk
 There is a total of           49  slices in each chunk
 There are            6  chunks
 There is a total of          294  slices in all the chunks

 There is a total of           84  messages to assemble faces between chunks

 Generating message            1  for faces out of           84
 ...
 Generating message           84  for faces out of           84

 all the messages for chunk faces have the right size

 Generating message            1  for corners out of            8
 ...
 Generating message            8  for corners out of            8

   ...preparing MPI interfaces

 outer core region:
   #max of points in MPI buffers along xi npoin2D_xi =          877
   #max of array elements transferred npoin2D_xi*NDIM =         2631

   #max of points in MPI buffers along eta npoin2D_eta =          877
   #max of array elements transferred npoin2D_eta*NDIM =         2631

 outer core MPI:
   maximum interfaces:           8
   MPI addressing maximum interfaces:           8
   MPI addressing : all interfaces okay

   total MPI interface points :      1106232
   unique MPI interface points:       954912
   maximum valence            :            3
   total assembled MPI interface points:      954912

   ...element inner/outer separation 
 percentage of edge elements in outer core    81.39535     %
 percentage of volume elements in outer core    18.60465     %

   ...element mesh coloring 
   mesh coloring:  F

   ...creating mass matrix

   ...saving binary files

 calculated region volume:   0.6531365    
                 top area:    3.749325    
             exact area:    3.74932546676467     
              bottom area:   0.4619356    
             exact area:   0.461935773543664     

 *******************************************
 creating mesh in region            3
 this region is the inner core
 *******************************************

 first pass

   ...allocating arrays 

   ...setting up layers 

   ...creating mesh elements 
   creating layer            1 out of            1
     number of regular elements  =            8
    100.0%    current clock (NOT elapsed) time is: 17h 24min 53sec
   layers done

   number of elements (per slice)        =            8
   total number of elements (all slices) =         2352

   creating central cube
   central cube done
     total number of elements =           64

   ...creating global addressing
     total number of points            :         8000
     array memory required per process :   0.183105468750000      MB
     getting global points             : npointot =         8000  nspec = 
          64
     creating indirect addressing
     ibool ok

   ...creating MPI buffers

 second pass

   ...allocating arrays 

   ...setting up layers 

   ...creating mesh elements 
   creating layer            1 out of            1
     number of regular elements  =            8
    100.0%    current clock (NOT elapsed) time is: 17h 24min 53sec
   layers done

   number of elements (per slice)        =            8
   total number of elements (all slices) =         2352

   creating central cube
   central cube done
     total number of elements =           64

   ...fills global mesh points 

   ...checking mesh resolution and time step

 ----------------------------------
   Verification of mesh parameters:
 ----------------------------------
   Region is inner core

   Min Vs =    3.504312      (km/s)
   Max Vp =    11.26220      (km/s)

   Max element edge size =    264.9278      (km)
   Min element edge size =    78.39577      (km)
   Max/min ratio =    3.379363    

   Max Jacobian eigenvalue ratio =    1.000000    
   Min Jacobian eigenvalue ratio =   0.3343573    

   Minimum period resolved =    40.02860      (s)
   Minimum period resolved (empirical) =    39.71247      (s)
   Maximum suggested time step =   0.6650000      (s)

   for DT :   0.2350000      (s)
   Max stability for wave velocities =   0.1938944    
 ----------------------------------

   ...precomputing Jacobian

   ...creating chunk buffers

 ----- creating chunk buffers -----

 There are            7  slices along xi in each chunk
 There are            7  slices along eta in each chunk
 There is a total of           49  slices in each chunk
 There are            6  chunks
 There is a total of          294  slices in all the chunks

 There is a total of           84  messages to assemble faces between chunks

 Generating message            1  for faces out of           84
 ...
 Generating message           84  for faces out of           84

 all the messages for chunk faces have the right size

 Generating message            1  for corners out of            8
 ...
 Generating message            8  for corners out of            8

   ...preparing MPI interfaces

 inner core region:
   #max of points in MPI buffers along xi npoin2D_xi =          585
   #max of array elements transferred npoin2D_xi*NDIM =         1755

   #max of points in MPI buffers along eta npoin2D_eta =          585
   #max of array elements transferred npoin2D_eta*NDIM =         1755

 including central cube
   number of messages in cube :            9
   number of 2D points in cube:          100
   creating central cube done

 inner core MPI:
 inner core with central cube MPI:
   maximum interfaces:          27
   MPI addressing maximum interfaces:          27
   MPI addressing : all interfaces okay

   total MPI interface points :       247410
   unique MPI interface points:       186886
   maximum valence            :            7
   total assembled MPI interface points:      186886

   ...element inner/outer separation 
 percentage of edge elements in inner core    62.50000     %
 percentage of volume elements in inner core    37.50000     %

   ...element mesh coloring 
   mesh coloring:  F

   ...creating mass matrix

   ...saving binary files

 calculated region volume:   2.9522007E-02
                 top area:   0.4619356    
             exact area:   0.461935773543664     
              bottom area:   0.2116964    
             more or less similar area (central cube):   0.177877903851686     

 calculated volume:    4.18879020416824     
      exact volume:    4.18879020478639     

 computed total Earth mass for this density model and mesh: 
      5.9741141E+24  kg
    (should be not too far from 5.974E+24 kg)

 average density for this density model and mesh: 
       5515.210      kg/m3
    (should be not too far from 5514 kg/m3)

 position of the center of mass of the globe for this density model and mesh: 
    x =  -0.6062207      km
    y =  -0.4331040      km
    z =  -0.5200786      km
    distance to center =   0.9086057      km

 Repartition of elements in regions:
 ----------------------------------

 number of elements in each slice      :         2140
 total number of elements in all slices:       629160

  - crust and mantle:    88.97196      %
  - outer core:    8.037383      %
  - inner core:    2.990654      %

 for some mesh statistics, see comments in file OUTPUT_FILES/values_from_mesher.
 h

 Load balancing = 100 % by definition

 the time step of the solver will be DT =   0.2350000    

 using single precision for the calculations

 smallest and largest possible floating-point numbers are:   1.1754944E-38
  3.4028235E+38

 Elapsed time for mesh generation and buffer creation in seconds = 
   1.940220    
 Elapsed time for mesh generation and buffer creation in hh:mm:ss =    0 h 00 m 01 s

 End of mesh generation

output_solver.txt for main branch: 

 ******************************
 **** Specfem3D MPI Solver ****
 ******************************

 Version: v8.1.0-1-gb6fa4b6b

 Planet: Earth

 There are          294  MPI processes
 Processes are numbered from 0 to          293

 There are          112  elements along xi in each chunk
 There are          112  elements along eta in each chunk

 There are            7  slices along xi in each chunk
 There are            7  slices along eta in each chunk
 There is a total of           49  slices in each chunk
 There are            6  chunks
 There is a total of          294  slices in all the chunks

 NDIM =            3

 NGLLX =            5
 NGLLY =            5
 NGLLZ =            5

 using single precision for the calculations

 smallest and largest possible floating-point numbers are:   1.1754944E-38
  3.4028235E+38

 model: 1D_transversely_isotropic_prem_onecrust
   no oceans
   no ellipticity
   no surface topography
   no self-gravitation
   no rotation
   no attenuation

   no 3-D lateral variations in the mantle
   no heterogeneities in the mantle
   no crustal variations
   using one layer only in crust
   incorporating transverse isotropy
   no inner-core anisotropy
   no general mantle anisotropy

 creating global slice addressing

 Spatial distribution of the slices
                       48   41   34   27   20   13    6
 ...
                      189  182  175  168  161  154  147

 mesh databases:
   reading in crust/mantle databases...
   reading in outer core databases...
   reading in inner core databases...
   reading in coupling surface databases...
   reading in MPI databases...
   for overlapping of communications with calculations:

   percentage of edge elements in crust/mantle    40.33614     %
   percentage of volume elements in crust/mantle    59.66386     %

   percentage of edge elements in outer core    81.39535     %
   percentage of volume elements in outer core    18.60465     %

   percentage of edge elements in inner core    62.50000     %
   percentage of volume elements in inner core    37.50000     %

 Elapsed time for reading mesh in seconds =   0.4267846    

 adjacency:
   total number of elements in this slice =         1904

   using kd-tree search radius =    536.118396321980      (km)

   maximum search elements                                      =          603
   maximum of actual search elements (after distance criterion) =          602

   estimated typical element size at surface =    89.3530660536633      (km)
   maximum distance between neighbor centers =    441.738280605293      (km)

   maximum neighbors found per element =           37 
 (should be 37 for globe meshes)
   total number of neighbors           =        41192

   Elapsed time for detection of neighbors in seconds =   7.656294900000005E-002

 kd-tree:
   total data points:         1904
   theoretical   number of nodes:         3802
                tree memory size:   0.1160278     MB
   actual        number of nodes:         3807
                tree memory size:   0.1161804     MB
   maximum depth   :           15
   creation timing :   4.9102306E-04 (s)

 sources:           1

 ********************
  locating sources
 ********************

 source #            1

   source located in slice          191
                  in element         1814

   using moment tensor source: 
     xi coordinate of source in that element:  -0.852883335572835     
     eta coordinate of source in that element:   0.616002807332196     
     gamma coordinate of source in that element:  -0.557379564444648     

   source time function:
     using (quasi) Heaviside source time function

     half duration:    2.20000000000000       seconds
     time shift:   0.000000000000000E+000  seconds

   magnitude of the source:
        scalar moment M0 =   1.058504638629420E+025  dyne-cm
     moment magnitude Mw =    5.98312870109856     

   original (requested) position of the source:

         latitude:    9.06000000000000     
        longitude:    126.810000000000     
            depth:    19.0000000000000       km

   position of the source that will be used:

         latitude:    9.06000000000001     
        longitude:    126.810000000000     
            depth:    18.9999999999998       km

   Error in location of the source:   1.7683077E-13  km

 maximum error in location of the sources:   1.7683077E-13  km

 Elapsed time for detection of sources in seconds =   3.898862999999975E-003

 End of source detection - done

 receivers:

 Total number of receivers =           25

 ********************
  locating receivers
 ********************

 reading receiver information...

 Stations sorted by epicentral distance:
 Station #    15:             US.__IU._DAV    epicentral distance:      2.322989 degrees
 ...
 Station #     5:             US.___G._PPT    epicentral distance:     86.683128 degrees

 Station #           1 : US.___G._CAN
        original latitude:   -35.31871    
       original longitude:    148.9963    
      epicentral distance:    48.86376    
   closest estimate found:   2.6465607E-12  km away
    in slice          133  in element         1775
    at xi,eta,gamma coordinates =   -1.03769323121867     
  0.164707494915945       0.999994751769902     
    at lat/lon =   -35.31871       148.9963    
 ...

 Station #          25 : US.__PS._TSK
        original latitude:    36.21080    
       original longitude:    140.1097    
      epicentral distance:    29.61530    
   closest estimate found:   1.5816225E-12  km away
    in slice          146  in element         1882
    at xi,eta,gamma coordinates =  -0.812875998752598     
  0.282536766484881        1.00001588060012     
    at lat/lon =    36.21080       140.1097    

 maximum error in location of all the receivers:   3.0229788E-12  km

 Elapsed time for receiver detection in seconds =   4.890291000000047E-003

 End of receiver detection - done

 found a total of           25  receivers in all slices
 this total is okay

 source arrays:
   number of sources is            1
   size of source array                 =   1.4305115E-03 MB
                                        =   1.3969839E-06 GB

 seismograms:
   seismograms written by all processes
   Total number of simulation steps (NSTEP)                       =        25700
   writing out seismograms at every NTSTEP_BETWEEN_OUTPUT_SEISMOS =        25700
   number of subsampling steps for seismograms                    =            1
   Total number of samples for seismograms                        =        25700

   maximum number of local receivers is            4  in slice          146
   size of maximum seismogram array       =    1.176453     MB
                                          =   1.1488795E-03 GB

 Total number of samples for seismograms =        25700

 Reference radius of the globe used is    6371.00000000000       km

 no oceans

 no ellipticity

 no surface topography

 no self-gravitation

 no rotation

 no attenuation

 preparing mass matrices
 preparing constants
 preparing gravity arrays
 preparing elastic element arrays
   crust/mantle transverse isotropic and isotropic elements
   tiso elements =          832
   iso elements  =         1072
   inner core isotropic elements
 preparing wavefields
   allocating wavefields
   initializing wavefields
 preparing number of runs
   number of runs    :            1
   number of this run:            1
   time stepping     : begin/end =            1 /       25700
 preparing optimized arrays
   without force vectorization
   using Deville products
   fusing arrays: 
     size of fused arrays =    9.183884     MB
                          =   8.9686364E-03 GB
   fused arrays done
   bandwidth test (STREAM TRIAD): 
      memory accesses =    4.488773     MB
      timing  min/max =   1.3387320E-03 s /   5.1272861E-03 s
      timing      avg =   3.3018945E-03 s
      bandwidth       =    1.327592     GB/s

 Elapsed time for preparing timerun in seconds =   0.2799606    

 time loop:

               scheme:         Newmark
            time step:   0.2350000      s
 number of time steps:        25700
   current time steps:            1  to        25700
 total simulated time:    100.5994      minutes
 start time          :  -3.300000      seconds

 All processes are synchronized before time loop

 Starting time iteration loop...

 Time step #            5
 Time:  -3.9333332E-02  minutes
 Max norm displacement vector U in solid in all slices for forward prop. (m) = 
  3.7323841E-06
 Max non-dimensional potential Ufluid in fluid in all slices for forward prop. =
    0.0000000E+00
 Elapsed time in seconds =   0.156743282000000     
 Elapsed time in hh:mm:ss =      0 h 00 m 00 s
 Mean elapsed time per time step in seconds =   3.134865640000002E-002
 Time steps done =            5  out of        25700
 Time steps remaining =        25695
 Estimated remaining time in seconds =    805.503726198001     
 Estimated remaining time in hh:mm:ss =      0 h 13 m 25 s
 Estimated total run time in seconds =    805.660469480001     
 Estimated total run time in hh:mm:ss =      0 h 13 m 25 s
 We have done   1.9455252E-02 % of that
 The run will finish approximately on (in local time): Mon Sep 16, 2024 17:46
 ************************************************************
 **** BEWARE: the above time estimates are not very reliable
 **** because fewer than 100 iterations have been performed
 ************************************************************
 ...

 Time step #        25700
 Time:    100.5994      minutes
 Max norm displacement vector U in solid in all slices for forward prop. (m) = 
  2.4129199E-02
 Max non-dimensional potential Ufluid in fluid in all slices for forward prop. =
    3.5934058E-14
 Elapsed time in seconds =    468.469017560000     
 Elapsed time in hh:mm:ss =      0 h 07 m 48 s
 Mean elapsed time per time step in seconds =   1.822836644202335E-002
 Time steps done =        25700  out of        25700
 Time steps remaining =            0
 Estimated remaining time in seconds =   0.000000000000000E+000
 Estimated remaining time in hh:mm:ss =      0 h 00 m 00 s
 Estimated total run time in seconds =    468.469017560000     
 Estimated total run time in hh:mm:ss =      0 h 07 m 48 s
 We have done    100.0000     % of that

 Total number of time steps written:        25700
 Writing the seismograms in parallel took   0.2587866      seconds

 Time-Loop Complete. Timing info:
 Total elapsed time in seconds =    468.740669439000     
 Total elapsed time in hh:mm:ss =      0 h 07 m 48 s

 finalizing simulation

 End of the simulation

output_mesher.txt for devel branch: 

 ****************************
 *** Specfem3D MPI Mesher ***
 ****************************

 Version: v8.1.0-144-g46dd8d68

 Planet: Earth

 There are          294  MPI processes
 Processes are numbered from 0 to          293

 There are          112  elements along xi in each chunk
 There are          112  elements along eta in each chunk

 There are            7  slices along xi in each chunk
 There are            7  slices along eta in each chunk
 There is a total of           49  slices in each chunk
 There are            6  chunks in the global mesh
 There is a total of          294  slices in the global mesh

 NGLLX =            5
 NGLLY =            5
 NGLLZ =            5

 Shape functions defined by NGNOD =           27  control nodes
 Surface shape functions defined by NGNOD2D =            9  control nodes

 model: 1D_transversely_isotropic_prem_onecrust
   no oceans
   no ellipticity
   no surface topography
   no self-gravitation
   no rotation
   no attenuation

   no 3-D lateral variations in the mantle
   no heterogeneities in the mantle
   no crustal variations
   using one layer only in crust
   incorporating transverse isotropy
   no inner-core anisotropy
   no general mantle anisotropy

 Reference radius of the globe used is    6371.00000000000       km

 Central cube is at a radius of    950.000000000000       km
 creating global slice addressing

 Spatial distribution of the slices
                       48   41   34   27   20   13    6
 ...
                      189  182  175  168  161  154  147

 additional mesh optimizations

 moho:
   no element stretching for 3-D moho surface

 internal topography 410/660:
   no element stretching for 3-D internal surfaces

 Radial Meshing parameters:
   NCHUNKS                =            6

   CENTER LAT/LON:            0.0000000E+00 /  0.0000000E+00
   GAMMA_ROTATION_AZIMUTH:    0.0000000E+00

   CHUNK WIDTH XI/ETA:         90.00000     /   90.00000    
   NEX XI/ETA:                       112 /         112

   NER_CRUST:                          1
   NER_80_MOHO:                        1
   NER_220_80:                         2
   NER_400_220:                        2
   NER_600_400:                        2
   NER_670_600:                        1
   NER_771_670:                        1
   NER_TOPDDOUBLEPRIME_771:           15
   NER_CMB_TOPDDOUBLEPRIME:            1
   NER_OUTER_CORE:                    16
   NER_TOP_CENTRAL_CUBE_ICB:           2
   SUPPRESS_CRUSTAL_MESH:    F

   R_CENTRAL_CUBE =    950.0000      km

 Mesh resolution:
   DT =   0.235000000000000     
   Minimum period =    39.71247      (s)

 *******************************************
 creating mesh in region            1
 this region is the crust and mantle
 *******************************************

 first pass

   ...allocating arrays 

   ...setting up layers 

   ...creating mesh elements 
   creating layer            1 out of           10
     number of regular elements  =            0
     number of doubling elements =          448
   setting tiso flags in mantle model
     10.0%    current clock (NOT elapsed) time is: 17h 24min 55sec
 ...
   creating layer           10 out of           10
     number of regular elements  =          256
    100.0%    current clock (NOT elapsed) time is: 17h 24min 55sec
   layers done

   number of elements (per slice)        =         1904
   total number of elements (all slices) =       559776

   ...creating global addressing
     total number of points            :       238000
     array memory required per process :    5.447388     MB
     getting global points             : npointot =       238000  nspec = 
        1904
     creating indirect addressing
     ibool ok

   ...creating MPI buffers

 second pass

   ...allocating arrays 

   ...setting up layers 

   ...creating mesh elements 
   creating layer            1 out of           10
     number of regular elements  =            0
     number of doubling elements =          448
     10.0%    current clock (NOT elapsed) time is: 17h 24min 55sec
 ...
   creating layer           10 out of           10
     number of regular elements  =          256
    100.0%    current clock (NOT elapsed) time is: 17h 24min 55sec
   layers done

   number of elements (per slice)        =         1904
   total number of elements (all slices) =       559776

   ...fills global mesh points 

   ...checking mesh resolution and time step

 ----------------------------------
   Verification of mesh parameters:
 ----------------------------------
   Region is crust/mantle

   Min Vs =    3.200000      (km/s)
   Max Vp =    13.71662      (km/s)

   Max element edge size =    255.1778      (km)
   Min element edge size =    18.53292      (km)
   Max/min ratio =    13.76890    

   Max Jacobian eigenvalue ratio =   0.9997538    
   Min Jacobian eigenvalue ratio =   7.1573265E-02

   Minimum period resolved =    26.30711      (s)
   Minimum period resolved (empirical) =    39.71247      (s)
   Maximum suggested time step =   0.2100000      (s)

   for DT :   0.2350000      (s)
   Max stability for wave velocities =   0.6014503    
 ----------------------------------

   saving vtk mesh files for resolution res_minimum_period...

   ...precomputing Jacobian

   ...creating chunk buffers

 ----- creating chunk buffers -----

 There are            7  slices along xi in each chunk
 There are            7  slices along eta in each chunk
 There is a total of           49  slices in each chunk
 There are            6  chunks
 There is a total of          294  slices in all the chunks

 There is a total of           84  messages to assemble faces between chunks

 Generating message            1  for faces out of           84
 ...
 Generating message           84  for faces out of           84

 all the messages for chunk faces have the right size

 Generating message            1  for corners out of            8
 ...
 Generating message            8  for corners out of            8

   ...preparing MPI interfaces

 crust/mantle region:
   #max of points in MPI buffers along xi npoin2D_xi =         3409
   #max of array elements transferred npoin2D_xi*NDIM =        10227

   #max of points in MPI buffers along eta npoin2D_eta =         3409
   #max of array elements transferred npoin2D_eta*NDIM =        10227

 crust mantle MPI:
   maximum interfaces:           8
   MPI addressing maximum interfaces:           8
   MPI addressing : all interfaces okay

   total MPI interface points :      4129944
   unique MPI interface points:      3885504
   maximum valence            :            3
   total unique MPI interface points:     3885504

   ...element inner/outer separation 

 for overlapping of communications with calculations:

 percentage of edge elements in crust/mantle    40.33614     %
 percentage of volume elements in crust/mantle    59.66386     %

   ...element mesh coloring 
      mesh coloring:  F

   ...creating mass matrix

   ...saving binary files

   ...saving mesh files

 calculated region volume:    3.506132    
                 top area:    12.56637    
               exact area:    12.5663706143592     
              bottom area:    3.749325    
               exact area:    3.74932546676467     

 *******************************************
 creating mesh in region            2
 this region is the outer core
 *******************************************

 first pass

   ...allocating arrays 

   ...setting up layers 

   ...creating mesh elements 
   creating layer            1 out of            2
     number of regular elements  =           28
     number of doubling elements =           32
     50.0%    current clock (NOT elapsed) time is: 17h 24min 57sec
   creating layer            2 out of            2
     number of regular elements  =          112
    100.0%    current clock (NOT elapsed) time is: 17h 24min 57sec
   layers done

   number of elements (per slice)        =          172
   total number of elements (all slices) =        50568

   ...creating global addressing
     total number of points            :        21500
     array memory required per process :   0.4920959     MB
     getting global points             : npointot =        21500  nspec = 
         172
     creating indirect addressing
     ibool ok

   ...creating MPI buffers

 second pass

   ...allocating arrays 

   ...setting up layers 

   ...creating mesh elements 
   creating layer            1 out of            2
     number of regular elements  =           28
     number of doubling elements =           32
     50.0%    current clock (NOT elapsed) time is: 17h 24min 57sec
   creating layer            2 out of            2
     number of regular elements  =          112
    100.0%    current clock (NOT elapsed) time is: 17h 24min 57sec
   layers done

   number of elements (per slice)        =          172
   total number of elements (all slices) =        50568

   ...fills global mesh points 

   ...checking mesh resolution and time step

 ----------------------------------
   Verification of mesh parameters:
 ----------------------------------
   Region is outer core

   Min Vp =    8.064794      (km/s)
   Max Vp =    10.35571      (km/s)

   Max element edge size =    263.6152      (km)
   Min element edge size =    70.57796      (km)
   Max/min ratio =    3.735093    

   Max Jacobian eigenvalue ratio =   0.9999157    
   Min Jacobian eigenvalue ratio =   0.1590487    

   Minimum period resolved =    21.87847      (s)
   Minimum period resolved (empirical) =    39.71247      (s)
   Maximum suggested time step =   0.6900000      (s)

   for DT :   0.2350000      (s)
   Max stability for wave velocities =   0.1862786    
 ----------------------------------

   saving vtk mesh files for resolution res_minimum_period...

   ...precomputing Jacobian

   ...creating chunk buffers

 ----- creating chunk buffers -----

 There are            7  slices along xi in each chunk
 There are            7  slices along eta in each chunk
 There is a total of           49  slices in each chunk
 There are            6  chunks
 There is a total of          294  slices in all the chunks

 There is a total of           84  messages to assemble faces between chunks

 Generating message            1  for faces out of           84
 ...
 Generating message           84  for faces out of           84

 all the messages for chunk faces have the right size

 Generating message            1  for corners out of            8
 ...
 Generating message            8  for corners out of            8

   ...preparing MPI interfaces

 outer core region:
   #max of points in MPI buffers along xi npoin2D_xi =          877
   #max of array elements transferred npoin2D_xi*NDIM =         2631

   #max of points in MPI buffers along eta npoin2D_eta =          877
   #max of array elements transferred npoin2D_eta*NDIM =         2631

 outer core MPI:
   maximum interfaces:           8
   MPI addressing maximum interfaces:           8
   MPI addressing : all interfaces okay

   total MPI interface points :      1106232
   unique MPI interface points:       954912
   maximum valence            :            3
   total assembled MPI interface points:      954912

   ...element inner/outer separation 
 percentage of edge elements in outer core    81.39535     %
 percentage of volume elements in outer core    18.60465     %

   ...element mesh coloring 
      mesh coloring:  F

   ...creating mass matrix

   ...saving binary files

   ...saving mesh files

 calculated region volume:   0.6531365    
                 top area:    3.749325    
               exact area:    3.74932546676467     
              bottom area:   0.4619356    
               exact area:   0.461935773543664     

 *******************************************
 creating mesh in region            3
 this region is the inner core
 *******************************************

 first pass

   ...allocating arrays 

   ...setting up layers 

   ...creating mesh elements 
   creating layer            1 out of            1
     number of regular elements  =            8
    100.0%    current clock (NOT elapsed) time is: 17h 24min 58sec
   layers done

   number of elements (per slice)        =            8
   total number of elements (all slices) =         2352

   creating central cube
   central cube done
     total number of elements =           64

   ...creating global addressing
     total number of points            :         8000
     array memory required per process :   0.1831055     MB
     getting global points             : npointot =         8000  nspec = 
          64
     creating indirect addressing
     ibool ok

   ...creating MPI buffers

 second pass

   ...allocating arrays 

   ...setting up layers 

   ...creating mesh elements 
   creating layer            1 out of            1
     number of regular elements  =            8
    100.0%    current clock (NOT elapsed) time is: 17h 24min 58sec
   layers done

   number of elements (per slice)        =            8
   total number of elements (all slices) =         2352

   creating central cube
   central cube done
     total number of elements =           64

   ...fills global mesh points 

   ...checking mesh resolution and time step

 ----------------------------------
   Verification of mesh parameters:
 ----------------------------------
   Region is inner core

   Min Vs =    3.504312      (km/s)
   Max Vp =    11.26220      (km/s)

   Max element edge size =    264.9278      (km)
   Min element edge size =    78.39577      (km)
   Max/min ratio =    3.379363    

   Max Jacobian eigenvalue ratio =    1.000000    
   Min Jacobian eigenvalue ratio =   0.3343573    

   Minimum period resolved =    40.02860      (s)
   Minimum period resolved (empirical) =    39.71247      (s)
   Maximum suggested time step =   0.6650000      (s)

   for DT :   0.2350000      (s)
   Max stability for wave velocities =   0.1938943    
 ----------------------------------

   saving vtk mesh files for resolution res_minimum_period...

   ...precomputing Jacobian

   ...creating chunk buffers

 ----- creating chunk buffers -----

 There are            7  slices along xi in each chunk
 There are            7  slices along eta in each chunk
 There is a total of           49  slices in each chunk
 There are            6  chunks
 There is a total of          294  slices in all the chunks

 There is a total of           84  messages to assemble faces between chunks

 Generating message            1  for faces out of           84
 ...
 Generating message           84  for faces out of           84

 all the messages for chunk faces have the right size

 Generating message            1  for corners out of            8
 ...
 Generating message            8  for corners out of            8

   ...preparing MPI interfaces

 inner core region:
   #max of points in MPI buffers along xi npoin2D_xi =          585
   #max of array elements transferred npoin2D_xi*NDIM =         1755

   #max of points in MPI buffers along eta npoin2D_eta =          585
   #max of array elements transferred npoin2D_eta*NDIM =         1755

 including central cube
   number of messages in cube :            9
   number of 2D points in cube:          100
   creating central cube done

 inner core MPI:
 inner core with central cube MPI:
   maximum interfaces:          27
   MPI addressing maximum interfaces:          27
   MPI addressing : all interfaces okay

   total MPI interface points :       247410
   unique MPI interface points:       186886
   maximum valence            :            7
   total assembled MPI interface points:      186886

   ...element inner/outer separation 
 percentage of edge elements in inner core    62.50000     %
 percentage of volume elements in inner core    37.50000     %

   ...element mesh coloring 
      mesh coloring:  F

   ...creating mass matrix

   ...saving binary files

   ...saving mesh files

 calculated region volume:   2.9522007E-02
                 top area:   0.4619356    
               exact area:   0.461935773543664     
              bottom area:   0.2116964    
               more or less similar area (central cube): 
  0.177877903851686     

 all mesh regions created - done

 *******************************************

 finalizing simulation:

 calculated volume:    4.18879020416824     
      exact volume:    4.18879020478639     

 computed total Earth mass for this density model and mesh: 
      5.9741141E+24  kg
    (should be not too far from 5.974E+24 kg)

 average density for this density model and mesh: 
       5515.210      kg/m3
    (should be not too far from 5514 kg/m3)

 position of the center of mass of the globe for this density model and mesh: 
    x =  -0.6062207      km
    y =  -0.4331040      km
    z =  -0.5200786      km
    distance to center =   0.9086057      km

 Repartition of elements in regions:
 ----------------------------------

 number of elements in each slice      :         2140
 total number of elements in all slices:       629160

  - crust and mantle       :    88.97196      %
  - outer core             :    8.037383      %
  - inner core             :    2.990654      %

 for some mesh statistics, see comments in file OUTPUT_FILES/values_from_mesher.
 h

 Load balancing = 100 % by definition

 the time step of the solver will be DT =   0.2350000    

 using single precision for the calculations

 smallest and largest possible floating-point numbers are:   1.1754944E-38
  3.4028235E+38

 Elapsed time for mesh generation and buffer creation in seconds = 
   3.786569    
 Elapsed time for mesh generation and buffer creation in hh:mm:ss =    0 h 00 m 03 s

 End of mesh generation

output_solver.txt for devel branch: 

 ******************************
 **** Specfem3D MPI Solver ****
 ******************************

 Version: v8.1.0-144-g46dd8d68

 Planet: Earth

 There are          294  MPI processes
 Processes are numbered from 0 to          293

 There are          112  elements along xi in each chunk
 There are          112  elements along eta in each chunk

 There are            7  slices along xi in each chunk
 There are            7  slices along eta in each chunk
 There is a total of           49  slices in each chunk
 There are            6  chunks
 There is a total of          294  slices in all the chunks

 NDIM =            3

 NGLLX =            5
 NGLLY =            5
 NGLLZ =            5

 using single precision for the calculations

 smallest and largest possible floating-point numbers are:   1.1754944E-38
  3.4028235E+38

 model: 1D_transversely_isotropic_prem_onecrust
   no oceans
   no ellipticity
   no surface topography
   no self-gravitation
   no rotation
   no attenuation

   no 3-D lateral variations in the mantle
   no heterogeneities in the mantle
   no crustal variations
   using one layer only in crust
   incorporating transverse isotropy
   no inner-core anisotropy
   no general mantle anisotropy

 creating global slice addressing

 Spatial distribution of the slices
                       48   41   34   27   20   13    6
 ...
                      189  182  175  168  161  154  147

 mesh databases:
   reading in crust/mantle databases...
   reading in outer core databases...
   reading in inner core databases...
   reading in coupling surface databases...
   reading in MPI databases...
   for overlapping of communications with calculations:

   percentage of edge elements in crust/mantle    40.33614     %
   percentage of volume elements in crust/mantle    59.66386     %

   percentage of edge elements in outer core    81.39535     %
   percentage of volume elements in outer core    18.60465     %

   percentage of edge elements in inner core    62.50000     %
   percentage of volume elements in inner core    37.50000     %

 Elapsed time for reading mesh in seconds =   0.4228267    

 adjacency:
   total number of elements in this slice =         1904

   using kd-tree search radius =    536.118396321980      (km)

   maximum search elements                                      =          603
   maximum of actual search elements (after distance criterion) =          602

   estimated typical element size at surface =    89.3530660536633      (km)
   maximum distance between neighbor centers =    441.738280605293      (km)

   maximum neighbors found per element =           37 
 (should be 37 for globe meshes)
   total number of neighbors           =        41192

   Elapsed time for detection of neighbors in seconds =   7.241178299999995E-002

 kd-tree:
   total data points:         1904
   theoretical   number of nodes:         3802
                tree memory size:   0.1160278     MB
   actual        number of nodes:         3807
                tree memory size:   0.1161804     MB
   maximum depth   :           15
   creation timing :   4.6098232E-04 (s)

 sources:           1

 ********************
  locating sources
 ********************

 source #            1

   source located in slice          191
                  in element         1814

   using moment tensor source:
     xi coordinate of source in that element:   -1.03486738298333     
     eta coordinate of source in that element:   0.616002722794674     
     gamma coordinate of source in that element:  -0.557378034139025     

     at (x,y,z) coordinates =  -0.589923600325576       0.788280283072162     
  0.156999070798417     

   source time function:
     using (quasi) Heaviside source time function

     half duration:    2.20000000000000       seconds
     time shift:   0.000000000000000E+000  seconds

   magnitude of the source:
        scalar moment M0 =   1.058504638629420E+025  dyne-cm
     moment magnitude Mw =    5.98312870109856     

   original (requested) position of the source:

         latitude:    9.06000000000000     
        longitude:    126.810000000000     
            depth:    19.0000000000000       km

   position of the source that will be used:

         latitude:    9.06000000000000     
        longitude:    126.810000000000     
            depth:    19.0000000000005       km

   Error in location of the source:   1.4146462E-12  km

 maximum error in location of the sources:   1.4146462E-12  km

 Elapsed time for detection of sources in seconds =   5.819685000000074E-003

 End of source detection - done

 receivers:

 Total number of receivers =           25

 ********************
  locating receivers
 ********************

 reading receiver information...

 Stations sorted by epicentral distance:
 Station #    15:             US.__IU._DAV    epicentral distance:      2.333758 degrees
 ...
 Station #     5:             US.___G._PPT    epicentral distance:     86.721901 degrees

 Station #           1 : US.___G._CAN
        original latitude:   -35.31871    
       original longitude:    148.9963    
      epicentral distance:    49.07474    
   closest estimate found:   7.9081126E-13  km away
    in slice          133  in element         1759
    at xi,eta,gamma coordinates =   0.494658344000912     
  0.164706518756413       0.999992594045171     
    at (x,y,z)                  =  -0.699377660929288     
  0.420289590670691      -0.578124162587018     
    at lat/lon                  =   -35.31871       148.9963    
 ...

 Station #          25 : US.__PS._TSK
        original latitude:    36.21080    
       original longitude:    140.1097    
      epicentral distance:    29.72227    
   closest estimate found:   1.0003060E-12  km away
    in slice          146  in element         1882
    at xi,eta,gamma coordinates =  -0.337021250532228     
  0.282538115161165        1.00001312279162     
    at (x,y,z)                  =  -0.619073981668109     
  0.517448223808016       0.590757768378481     
    at lat/lon                  =    36.21080       140.1097    

 maximum error in location of all the receivers:   4.2448592E-12  km

 Elapsed time for receiver detection in seconds =   6.232612999999887E-003

 End of receiver detection - done

 found a total of           25  receivers in all slices
 this total is okay

 source arrays:
   number of sources is            1
   size of source array                 =   1.4305115E-03 MB
                                        =   1.3969839E-06 GB

 seismograms:
   seismograms written by all processes
   Total number of simulation steps (NSTEP)                       =        25700
   writing out seismograms at every NTSTEP_BETWEEN_OUTPUT_SEISMOS =        25700
   number of subsampling steps for seismograms                    =            1
   Total number of samples for seismograms                        =        25700

   maximum number of local receivers is            4  in slice          146
   size of maximum seismogram array       =    1.176453     MB
                                          =   1.1488795E-03 GB

 Total number of samples for seismograms =        25700

 Reference radius of the globe used is    6371.00000000000       km

 no oceans

 no ellipticity

 no surface topography

 no self-gravitation

 no rotation

 no attenuation

 preparing mass matrices
 preparing constants
 preparing gravity arrays
 preparing elastic element arrays
   crust/mantle transverse isotropic and isotropic elements
   tiso elements =       244608
   iso elements  =       315168
   inner core isotropic elements
 preparing wavefields
   allocating wavefields
   initializing wavefields
 preparing number of runs
   number of runs    :            1
   number of this run:            1
   time stepping     : begin/end =            1 /       25700
 preparing optimized arrays
   without force vectorization
   using Deville products
   fusing arrays: 
     size of fused arrays =    9.183884     MB
                          =   8.9686364E-03 GB
   fused arrays done
   bandwidth test (STREAM TRIAD): 
      memory accesses =    4.488773     MB
      timing  min/max =   7.4690598E-04 s /   5.2978829E-03 s
      timing      avg =   2.4200636E-03 s
      bandwidth       =    1.811344     GB/s

 Elapsed time for preparing timerun in seconds =   0.2849818    

 time loop:

               scheme:         Newmark
            time step:   0.2350000      s
 number of time steps:        25700
   current time steps:            1  to        25700
 total simulated time:    100.5994      minutes
 start time          :  -3.300000      seconds

 All processes are synchronized before time loop

 Starting time iteration loop...

 Time step #            5
 Time:  -3.9333332E-02  minutes
 Max norm displacement vector U in solid in all slices for forward prop. (m) = 
  1.0674171E-05
 Max non-dimensional potential Ufluid in fluid in all slices for forward prop. =
    0.0000000E+00
 Elapsed time in seconds =   0.165974451000000     
 Elapsed time in hh:mm:ss =      0 h 00 m 00 s
 Mean elapsed time per time step in seconds =   3.319489019999997E-002
 Time steps done =            5  out of        25700
 Time steps remaining =        25695
 Estimated remaining time in seconds =    852.942703688999     
 Estimated remaining time in hh:mm:ss =      0 h 14 m 12 s
 Estimated total run time in seconds =    853.108678139999     
 Estimated total run time in hh:mm:ss =      0 h 14 m 13 s
 We have done   1.9455252E-02 % of that
 The run will finish approximately on (in local time): Mon Sep 16, 2024 17:47
 ************************************************************
 **** BEWARE: the above time estimates are not very reliable
 **** because fewer than 100 iterations have been performed
 ************************************************************
 ...

 Time step #        25700
 Time:    100.5994      minutes
 Max norm displacement vector U in solid in all slices for forward prop. (m) = 
  5.2489478E-02
 Max non-dimensional potential Ufluid in fluid in all slices for forward prop. =
    3.6080866E-14
 Elapsed time in seconds =    475.099044364000     
 Elapsed time in hh:mm:ss =      0 h 07 m 55 s
 Mean elapsed time per time step in seconds =   1.848634413867704E-002
 Time steps done =        25700  out of        25700
 Time steps remaining =            0
 Estimated remaining time in seconds =   0.000000000000000E+000
 Estimated remaining time in hh:mm:ss =      0 h 00 m 00 s
 Estimated total run time in seconds =    475.099044364000     
 Estimated total run time in hh:mm:ss =      0 h 07 m 55 s
 We have done    100.0000     % of that

 Writing the seismograms
 Total number of time steps written:        25700
 Writing the seismograms in parallel took   0.2870277      seconds

 Time-Loop Complete. Timing info:
 Total elapsed time in seconds =    475.402083366000     
 Total elapsed time in hh:mm:ss =      0 h 07 m 55 s

 finalizing simulation

 End of the simulation

OS

No response

danielpeter commented 1 month ago

hi Dorian,

i think these changes are expected as version 8.1 deals slightly different with projecting geographic positions onto a geocentric framework. version 8.0 by default "corrects" geographic positions even for a spherical mesh. version 8.1 only converts geographic to geocentric if the mesh is actually elliptic. given your example above uses a spherical mesh, the positioning of source and receivers will be slightly different in version 8.0 and 8.1.

as a general suggestion, use the most recent version for GLOBE, i.e. version 8.1.0. this newest version will improve outputs, in particular how ellipticity and topography are incorporated when positioning sources & receivers. thus, for any code development, start from the latest devel branch version.

dosoe commented 1 month ago

Ok, but I have a problem then: I am integrating semucb into specfem and I check the results by comparing it with our 'homegrown' fork of specfem3d_globe where we have integrated semucb. However, checking if I did so correctly relies on running the same model in both codes and checking if the results match. Right now the results don't match even when running prem, so I cannot use this method to check whether I integrated semucb correctly. Should I run my tests with the ellipticity enabled, so that both use an elliptic mesh? So far I have disabled all additional factors ( ellipticity, rotation etc) to make it simpler to diagnose problems should they arise.

dosoe commented 1 month ago

Running the same tests with ellipticity enabled gives a perfect fit. I believes that settles this.