Closed lsawade closed 4 years ago
I must have done something wrong with the master
branch. The master
branch does compile.
hi Lucas, what gcc version were you using?
could you provide the output for
gfortran -v
Hi all,
Working on this on traverse. Disabling optimization ( -O0) with gcc makes it work... Seems like an aggresive optimization problem. Valgrind also makes it pass without problem...
FYI, the crash happens when meshing region 2 (liquid outer core).
gfortran -v gfortran -v Utilisation des specs internes. COLLECT_GCC=gfortran COLLECT_LTO_WRAPPER=/usr/libexec/gcc/ppc64le-redhat-linux/4.8.5/lto-wrapper Target: ppc64le-redhat-linux Configuré avec: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-ppc64le-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-ppc64le-redhat-linux/cloog-install --enable-gnu-indirect-function --enable-secureplt --with-long-double-128 --enable-targets=powerpcle-linux --disable-multilib --with-cpu-64=power8 --with-tune-64=power8 --build=ppc64le-redhat-linux Modèle de thread: posix gcc version 4.8.5 20150623 (Red Hat 4.8.5-37) (GCC)
It works also with -O1.
Update on GDB output:
Program received signal SIGFPE, Arithmetic exception.
0x000000001009df94 in get_model (iregion_code=2, ispec=1, nspec=
Well problem is solved...
Actually the problem is here: lines 169-170 of get_model.F90 vs = sqrt(((1.d0-2.d0eta_aniso)vphvph + vpvvpv &
The square root argument might become slightly negative because of roundoff errors. Those may be compiler and architecture dependents. If it is slightly negative result of sqrt(-something) is complex. Some compilers trhow errors, gives zero or gives NaN. Here we compute vs in the fluid region thus it should be zero but not exactly because of roundoff errors.
The trick is to modify so that: vs = sqrt(abs((1.d0-2.d0eta_aniso)vphvph + vpvvpv &
Another possibility is to take max(argument, 0).
Stephen
hi Stephen, great spotting this one!
what happens if you use instead:
if (iregion_code == IREGION_OUTER_CORE) then
! fluid with zero shear speed
vs = 0.d0
else
vs = sqrt(((1.d0-2.d0*eta_aniso)*vph*vph + vpv*vpv &
+ 5.d0*vsh*vsh + (6.d0+4.d0*eta_aniso)*vsv*vsv)/15.d0)
endif
that is a bit more explicit when reading through.
I agree this would be more explicit, should I make this substitution?
yes, that would be great - thanks. -daniel
On 19 Jun 2020, at 8:38 PM, 1sbeller notifications@github.com wrote:
I agree this would be more explicit, should I make this substitution?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/geodynamics/specfem3d_globe/issues/690#issuecomment-646810253, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABMLXWMJ6ZJRYEF6HJUM5KTRXOWBNANCNFSM4OCI7KMA.
Wow! This is great thanks both to you @danielpeter and you @1sbeller !
I think we are ready to close this one with #693
Thanks, @1sbeller!
Hi!
I'm having trouble meshing the most recent version (
master
/devel
) on aPower9
machine at Princeton. Details below. I'm still trying to backtrace the exact issue. The issue cannot be replicated onSummit
.Hardware and last good commit (as far as I know)
Over the past half year I have been using the Traverse cluster at Princeton (Traverse). I wasn't up to date with the repo and the last commit I was working with was
This commit compiles without issue using
ADIOS 1.13.1
.Error
Using the most recent
master
ordevel
, the error is apparently thrown in thespecfem3d_globe/src/meshfem3D/meshfem3D_models.F90
routine in Line352
which calls . I'm pasting bothoutput_mesher.txt
as well as thestdout
below.Note that the issue is seemingly not related to the dimensions of the velocity model (
1D_transverse_isotropic_prem
here) since I tried compilings362ani
as well, without luck. It might have to do with background values(?).
``` $ specfem3d_globe/OUTPUT_FILES/output_mesher.txt **************************** *** Specfem3D MPI Mesher *** **************************** Version: v7.0.2-237-gd491b15 Planet: Earth There are 6 MPI processes Processes are numbered from 0 to 5 There are 128 elements along xi in each chunk There are 128 elements along eta in each chunk There are 1 slices along xi in each chunk There are 1 slices along eta in each chunk There is a total of 1 slices in each chunk There are 6 chunks in the global mesh There is a total of 6 slices in the global mesh NGLLX = 5 NGLLY = 5 NGLLZ = 5 Shape functions defined by NGNOD = 27 control nodes Surface shape functions defined by NGNOD2D = 9 control nodes model: 1D_transversely_isotropic_prem incorporating the oceans using equivalent load incorporating ellipticity incorporating surface topography incorporating self-gravitation (Cowling approximation) incorporating rotation incorporating attenuation using 3 standard linear solids no 3-D lateral variations in the mantle no heterogeneities in the mantle no crustal variations using unmodified 1D crustal model with two layers incorporating transverse isotropy no inner-core anisotropy no general mantle anisotropy Reference radius of the globe used is 6371.0000000000000 km Central cube is at a radius of 950.00000000000000 km creating global slice addressing Spatial distribution of the slices 0 2 1 4 5 3 incorporating topography topo file : DATA/topo_bathy/topo_bathy_etopo4_smoothed_window_7.bin resolution in minutes: 4.00000000 topography/bathymetry: min/max = -7747 5507 Elapsed time for reading in seconds = 0.13036037500000003 additional mesh optimizations moho: no element stretching for 3-D moho surface internal topography 410/660: no element stretching for 3-D internal surfaces Radial Meshing parameters: NCHUNKS = 6 CENTER LAT/LON: 0.00000000 / 0.00000000 GAMMA_ROTATION_AZIMUTH: 0.00000000 CHUNK WIDTH XI/ETA: 90.0000000 / 90.0000000 NEX XI/ETA: 128 / 128 NER_CRUST: 2 NER_80_MOHO: 1 NER_220_80: 2 NER_400_220: 2 NER_600_400: 2 NER_670_600: 1 NER_771_670: 1 NER_TOPDDOUBLEPRIME_771: 15 NER_CMB_TOPDDOUBLEPRIME: 1 NER_OUTER_CORE: 16 NER_TOP_CENTRAL_CUBE_ICB: 2 SUPPRESS_CRUSTAL_MESH: F R_CENTRAL_CUBE = 950.000000 km Mesh resolution: DT = 0.19000000000000000 Minimum period = 34.7484131 (s) MIN_ATTENUATION_PERIOD = 34.7484131 MAX_ATTENUATION_PERIOD = 1954.04700 ******************************************* creating mesh in region 1 this region is the crust and mantle ******************************************* first pass ...allocating arrays ...setting up layers ...creating mesh elements creating layer 1 out of 11 setting tiso flags in mantle model 9.1% current clock (NOT elapsed) time is: 10h 57min 54sec creating layer 2 out of 11 18.2% current clock (NOT elapsed) time is: 10h 57min 54sec creating layer 3 out of 11 27.3% current clock (NOT elapsed) time is: 10h 57min 54sec creating layer 4 out of 11 36.4% current clock (NOT elapsed) time is: 10h 57min 55sec creating layer 5 out of 11 45.5% current clock (NOT elapsed) time is: 10h 57min 55sec creating layer 6 out of 11 54.5% current clock (NOT elapsed) time is: 10h 57min 56sec creating layer 7 out of 11 63.6% current clock (NOT elapsed) time is: 10h 57min 56sec creating layer 8 out of 11 72.7% current clock (NOT elapsed) time is: 10h 57min 56sec creating layer 9 out of 11 81.8% current clock (NOT elapsed) time is: 10h 57min 56sec creating layer 10 out of 11 90.9% current clock (NOT elapsed) time is: 10h 57min 57sec creating layer 11 out of 11 100.0% current clock (NOT elapsed) time is: 10h 57min 58sec ...creating global addressing ...creating MPI buffers second pass ...allocating arrays ...setting up layers ...creating mesh elements creating layer 1 out of 11 9.1% current clock (NOT elapsed) time is: 10h 58min 58sec creating layer 2 out of 11 18.2% current clock (NOT elapsed) time is: 10h 59min 01sec creating layer 3 out of 11 27.3% current clock (NOT elapsed) time is: 10h 59min 01sec creating layer 4 out of 11 36.4% current clock (NOT elapsed) time is: 10h 59min 06sec creating layer 5 out of 11 45.5% current clock (NOT elapsed) time is: 10h 59min 15sec creating layer 6 out of 11 54.5% current clock (NOT elapsed) time is: 10h 59min 16sec creating layer 7 out of 11 63.6% current clock (NOT elapsed) time is: 10h 59min 17sec creating layer 8 out of 11 72.7% current clock (NOT elapsed) time is: 10h 59min 20sec creating layer 9 out of 11 81.8% current clock (NOT elapsed) time is: 10h 59min 23sec creating layer 10 out of 11 90.9% current clock (NOT elapsed) time is: 10h 59min 28sec creating layer 11 out of 11 100.0% current clock (NOT elapsed) time is: 10h 59min 34sec ...fills global mesh points ...checking mesh resolution and time step ---------------------------------- Verification of mesh parameters: ---------------------------------- Region is crust/mantle Min Vs = 3.19999981 (km/s) Max Vp = 13.7166214 (km/s) Max element edge size = 223.495636 (km) Min element edge size = 9.07211971 (km) Max/min ratio = 24.6354370 Max Jacobian eigenvalue ratio = 0.998868346 Min Jacobian eigenvalue ratio = 6.10016361E-02 Minimum period resolved = 26.3479710 (s) Minimum period resolved (empirical) = 34.7484131 (s) Maximum suggested time step = 0.125000000 (s) for DT : 0.189999998 (s) Max stability for wave velocities = 0.824762464 ---------------------------------- ...precomputing Jacobian ...creating chunk buffers ----- creating chunk buffers ----- There are 1 slices along xi in each chunk There are 1 slices along eta in each chunk There is a total of 1 slices in each chunk There are 6 chunks There is a total of 6 slices in all the chunks There is a total of 12 messages to assemble faces between chunks Generating message 1 for faces out of 12 Generating message 2 for faces out of 12 Generating message 3 for faces out of 12 Generating message 4 for faces out of 12 Generating message 5 for faces out of 12 Generating message 6 for faces out of 12 Generating message 7 for faces out of 12 Generating message 8 for faces out of 12 Generating message 9 for faces out of 12 Generating message 10 for faces out of 12 Generating message 11 for faces out of 12 Generating message 12 for faces out of 12 all the messages for chunk faces have the right size Generating message 1 for corners out of 8 Generating message 2 for corners out of 8 Generating message 3 for corners out of 8 Generating message 4 for corners out of 8 Generating message 5 for corners out of 8 Generating message 6 for corners out of 8 Generating message 7 for corners out of 8 Generating message 8 for corners out of 8 ...preparing MPI interfaces crust/mantle region: #max of points in MPI buffers along xi npoin2D_xi = 28589 #max of array elements transferred npoin2D_xi*NDIM = 85767 #max of points in MPI buffers along eta npoin2D_eta = 28589 #max of array elements transferred npoin2D_eta*NDIM = 85767 crust mantle MPI: maximum interfaces: 4 MPI addressing maximum interfaces: 4 MPI addressing : all interfaces okay total MPI interface points : 686136 unique MPI interface points: 683520 maximum valence : 2 total unique MPI interface points: 683520 ...element inner/outer separation for overlapping of communications with calculations: percentage of edge elements in crust/mantle 5.41955996 % percentage of volume elements in crust/mantle 94.5804367 % ...element mesh coloring mesh coloring: F ...creating mass matrix updates mass matrix with ocean load ...saving binary files solver in ADIOS 1 file format saving arrays in ADIOS file: ./DATABASES_MPI/solver_data.bp saving arrays in ADIOS file: ./DATABASES_MPI/boundary.bp saving arrays in ADIOS file: ./DATABASES_MPI/attenuation.bp MPI in ADIOS 1 file format saving arrays in ADIOS file: ./DATABASES_MPI/solver_data_mpi.bp calculated region volume: 3.50144863 top area: 12.5574989 bottom area: 3.74933386 ******************************************* creating mesh in region 2 this region is the outer core ******************************************* first pass ...allocating arrays ...setting up layers ...creating mesh elements creating layer 1 out of 2 50.0% current clock (NOT elapsed) time is: 11h 00min 12sec creating layer 2 out of 2 100.0% current clock (NOT elapsed) time is: 11h 00min 13sec ...creating global addressing ...creating MPI buffers second pass ...allocating arrays ...setting up layers ...creating mesh elements creating layer 1 out of 2 ```output_mesher.txt
Error
```bash [traverse-k02g2:76669:0:76669] Caught signal 8 (Floating point exception: floating-point invalid operation) [traverse-k02g2:76670:0:76670] Caught signal 8 (Floating point exception: floating-point invalid operation) [traverse-k02g2:76671:0:76671] Caught signal 8 (Floating point exception: floating-point invalid operation) [traverse-k02g2:76672:0:76672] Caught signal 8 (Floating point exception: floating-point invalid operation) [traverse-k02g2:76673:0:76673] Caught signal 8 (Floating point exception: floating-point invalid operation) [traverse-k02g2:76674:0:76674] Caught signal 8 (Floating point exception: floating-point invalid operation) ==== backtrace (tid: 76670) ==== 0 0x00000000100c3c2c meshfem3d_models_get1d_val_() /scratch/gpfs/lsawade/MagicScripts/specfem3d_globe/src/meshfem3D/meshfem3D_models.F90:352 1 0x000000001009def0 get_model_() /scratch/gpfs/lsawade/MagicScripts/specfem3d_globe/src/meshfem3D/get_model.F90:165 2 0x000000001006c940 compute_element_properties_() /scratch/gpfs/lsawade/MagicScripts/specfem3d_globe/src/meshfem3D/compute_element_properties.f90:149 3 0x0000000010095468 create_regular_elements_() /scratch/gpfs/lsawade/MagicScripts/specfem3d_globe/src/meshfem3D/create_regular_elements.f90:277 4 0x00000000100944ec create_regions_elements_() /scratch/gpfs/lsawade/MagicScripts/specfem3d_globe/src/meshfem3D/create_regions_elements.f90:141 5 0x0000000010090cb0 create_regions_mesh_() /scratch/gpfs/lsawade/MagicScripts/specfem3d_globe/src/meshfem3D/create_regions_mesh.F90:176 6 0x00000000100819d8 create_meshes_() /scratch/gpfs/lsawade/MagicScripts/specfem3d_globe/src/meshfem3D/create_meshes.f90:159 7 0x0000000010005658 xmeshfem3d() /scratch/gpfs/lsawade/MagicScripts/specfem3d_globe/src/meshfem3D/meshfem3D.f90:377 8 0x0000000010005658 main() /scratch/gpfs/lsawade/MagicScripts/specfem3d_globe/src/meshfem3D/meshfem3D.f90:385 9 0x0000000000025200 generic_start_main.isra.0() libc-start.c:0 10 0x00000000000253f4 __libc_start_main() ???:0 . . . [same for the other workers] . . . ================================= srun: error: traverse-k02g2: tasks 0-5: Floating point exception srun: Terminating job step 78840.0 ```stdout
/stderr
Configuration & Compilation
Below, I'll past all the modules used etc. and how I flag the compilation
Par_file
``` SIMULATION_TYPE = 1 # set to 1 for forward simulations, 2 for adjoint simulations for sources, and 3 for kernel simulations NOISE_TOMOGRAPHY = 0 # flag of noise tomography, three steps (1,2,3). If earthquake simulation, set it to 0. SAVE_FORWARD = .false. # save last frame of forward simulation or not NCHUNKS = 6 ANGULAR_WIDTH_XI_IN_DEGREES = 20.d0 # angular size of a chunk ANGULAR_WIDTH_ETA_IN_DEGREES = 20.d0 CENTER_LATITUDE_IN_DEGREES = 40.d0 CENTER_LONGITUDE_IN_DEGREES = 25.d0 GAMMA_ROTATION_AZIMUTH = 0.d0 NEX_XI = 128 NEX_ETA = 128 NPROC_XI = 1 NPROC_ETA = 1 # transversely_isotropic_prem_plus_3D_crust_2.0, 3D_anisotropic, 3D_attenuation, # s20rts, s40rts, s362ani, s362iso, s362wmani, s362ani_prem, s362ani_3DQ, s362iso_3DQ, # s29ea, sea99_jp3d1994, sea99, jp3d1994, heterogen, full_sh, sgloberani_aniso, sgloberani_iso MODEL = 1D_transversely_isotropic_prem OCEANS = .true. ELLIPTICITY = .true. TOPOGRAPHY = .true. GRAVITY = .true. ROTATION = .true. ATTENUATION = .true. ABSORBING_CONDITIONS = .false. RECORD_LENGTH_IN_MINUTES = 10.d0 PARTIAL_PHYS_DISPERSION_ONLY = .false. UNDO_ATTENUATION = .false. MEMORY_INSTALLED_PER_CORE_IN_GB = 4.d0 PERCENT_OF_MEM_TO_USE_PER_CORE = 85.d0 EXACT_MASS_MATRIX_FOR_ROTATION = .false. USE_LDDRK = .false. INCREASE_CFL_FOR_LDDRK = .true. RATIO_BY_WHICH_TO_INCREASE_IT = 1.5d0 MOVIE_SURFACE = .false. MOVIE_VOLUME = .false. MOVIE_COARSE = .false. NTSTEP_BETWEEN_FRAMES = 50 HDUR_MOVIE = 0.d0 MOVIE_VOLUME_TYPE = 2 MOVIE_TOP_KM = -100.0 MOVIE_BOTTOM_KM = 1000.0 MOVIE_WEST_DEG = -90.0 MOVIE_EAST_DEG = 90.0 MOVIE_NORTH_DEG = 90.0 MOVIE_SOUTH_DEG = -90.0 MOVIE_START = 0 MOVIE_STOP = 40000 SAVE_MESH_FILES = .false. NUMBER_OF_RUNS = 1 NUMBER_OF_THIS_RUN = 1 LOCAL_PATH = ./DATABASES_MPI LOCAL_TMP_PATH = ./DATABASES_MPI NTSTEP_BETWEEN_OUTPUT_INFO = 500 NTSTEP_BETWEEN_OUTPUT_SEISMOS = 5000000 NTSTEP_BETWEEN_READ_ADJSRC = 1000 USE_FORCE_POINT_SOURCE = .false. SAVE_SEISMOGRAMS_STRAIN = .false. SAVE_SEISMOGRAMS_IN_ADJOINT_RUN = .false. OUTPUT_SEISMOS_ASCII_TEXT = .false. OUTPUT_SEISMOS_SAC_ALPHANUM = .false. OUTPUT_SEISMOS_SAC_BINARY = .true. OUTPUT_SEISMOS_ASDF = .false. ROTATE_SEISMOGRAMS_RT = .false. WRITE_SEISMOGRAMS_BY_MASTER = .false. SAVE_ALL_SEISMOS_IN_ONE_FILE = .false. USE_BINARY_FOR_LARGE_FILE = .false. RECEIVERS_CAN_BE_BURIED = .true. PRINT_SOURCE_TIME_FUNCTION = .false. READ_ADJSRC_ASDF = .false. ANISOTROPIC_KL = .false. SAVE_TRANSVERSE_KL_ONLY = .false. SAVE_AZIMUTHAL_ANISO_KL_ONLY = .false. APPROXIMATE_HESS_KL = .false. USE_FULL_TISO_MANTLE = .false. SAVE_SOURCE_MASK = .false. SAVE_REGULAR_KL = .false. NUMBER_OF_SIMULTANEOUS_RUNS = 1 BROADCAST_SAME_MESH_AND_MODEL = .false. GPU_MODE = .true. GPU_RUNTIME = 1 GPU_PLATFORM = NVIDIA GPU_DEVICE = Tesla ADIOS_ENABLED = .true. ADIOS_FOR_FORWARD_ARRAYS = .true. ADIOS_FOR_MPI_ARRAYS = .true. ADIOS_FOR_ARRAYS_SOLVER = .true. ADIOS_FOR_SOLVER_MESHFILES = .true. ADIOS_FOR_AVS_DX = .true. ADIOS_FOR_KERNELS = .true. ADIOS_FOR_MODELS = .true. ADIOS_FOR_UNDO_ATTENUATION = .true. ```Modules, Compiler & Packages
```bash module load openmpi/gcc/4.0.3rc1/64 module load cudatoolkit/10.0 # C/C++ compiler CC=gcc CXX=g++ MPICC=mpicc FCFLAGS="-g" \# Fortran compiler FC=gfortran MPIFC=mpif90 CFLAGS="" # CUDA (here CUDA 5 because my GPU cannot support more, poor boy) CUDA_WITH="--with-cuda=cuda8" ADIOS_LINK="https://users.nccs.gov/~pnorbert/adios-1.13.1.tar.gz" ADIOS_DESTDIR="${ADIOS_DIR}/build" ADIOS_WITH="--with-adios" ADIOS_CONFIG="$ADIOS_DESTDIR/bin/adios_config" ```Configuration
```bash ./configure CC=$CC CXX=$CXX FC=$FC MPIFC=$MPIFC \ CFLAGS="$CFLAGS" FCLAGS="$FCFLAGS" \ $CUDA_WITH CUDA_LIB="$CUDA_LIB" \ $ADIOS_WITH ADIOS_CONFIG="$ADIOS_CONFIG" ```