search

geodynamics / aspect

A parallel, extensible finite element code to simulate convection in both 2D and 3D models.
https://aspect.geodynamics.org/
Other
224 stars 235 forks source link

Deformed cell for 3D mesh deformation in release mode #5980

Open anne-glerum opened 1 month ago

anne-glerum commented 1 month ago

@Kaili270 and I ran into a curious error that only occurs in Release mode, only in 3D, and only for certain combinations of X mesh extent and X mesh repetitions.

dealii version: 9.5.2 ASPECT commit (May 29 2024): 249618d20bb5c8601226c4bf48632883d4029558

Error message:

-----------------------------------------------------------------------------
--     . version 2.6.0-pre (lithosphere_with_rift_IC_cookbook, bd49ff69a)
--     . using deal.II 9.5.2
--     .       with 32 bit indices and vectorization level 3 (512 bits)
--     . using Trilinos 13.2.0
--     . using p4est 2.3.2
--     . using Geodynamic World Builder 0.6.0
--     . running in OPTIMIZED mode
--     . running with 1 MPI process
-----------------------------------------------------------------------------

-----------------------------------------------------------------------------
-- For information on how to cite ASPECT, see:
--   https://aspect.geodynamics.org/citing.html?ver=2.6.0-pre&sha=bd49ff69a&src=code
-----------------------------------------------------------------------------
Number of active cells: 20 (on 1 levels)
Number of degrees of freedom: 1,248 (891+60+297)

Number of mesh deformation degrees of freedom: 180
---------------------------------------------------------
TimerOutput objects finalize timed values printed to the
screen by communicating over MPI in their destructors.
Since an exception is currently uncaught, this
synchronization (and subsequent output) will be skipped
to avoid a possible deadlock.
---------------------------------------------------------

----------------------------------------------------
Exception 'ExcMessage( "The Newton iteration to find the reference point " "did not converge in 10 iterations. Do you have a " "deformed cell? (See the glossary for a definition " "of what a deformed cell is. You may want to output " "the vertices of your cell.")' on rank 0 on processing: 

--------------------------------------------------------
An error occurred in line <977> of file </home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/source/grid/manifold.cc> in function
    dealii::Tensor<1, spacedim> dealii::FlatManifold<dim, spacedim>::normal_vector(const typename dealii::Triangulation<dim, spacedim>::face_iterator&, const dealii::Point<spacedim>&) const [with int dim = 3; int spacedim = 3; typename dealii::Triangulation<dim, spacedim>::face_iterator = dealii::TriaIterator<dealii::TriaAccessor<2, 3, 3> >]
The violated condition was: 
    iteration < 10
Additional information: 
    The Newton iteration to find the reference point did not converge in
    10 iterations. Do you have a deformed cell? (See the glossary for a
    definition of what a deformed cell is. You may want to output the
    vertices of your cell.

Stacktrace:
-----------
#0  /home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/deal.II-v9.5.2/lib/libdeal_II.so.9.5.2: dealii::FlatManifold<3, 3>::normal_vector(dealii::TriaIterator<dealii::TriaAccessor<2, 3, 3> > const&, dealii::Point<3, double> const&) const
#1  /home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/deal.II-v9.5.2/lib/libdeal_II.so.9.5.2: void dealii::VectorTools::internal::map_dofs_to_normal_vectors_and_normal_fluxes<3, 3>(dealii::DoFHandler<3, 3>::cell_iterator const&, unsigned int, std::set<unsigned int, std::less<unsigned int>, std::allocator<unsigned int> > const&, std::map<unsigned int, dealii::Function<3, double> const*, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, dealii::Function<3, double> const*> > > const&, dealii::hp::FEFaceValues<3, 3>&, unsigned int, dealii::IndexSet const&, unsigned int, std::multimap<dealii::VectorTools::internal::VectorDoFTuple<3>, std::pair<dealii::Tensor<1, 3, double>, dealii::DoFHandler<3, 3>::cell_iterator>, std::less<dealii::VectorTools::internal::VectorDoFTuple<3> >, std::allocator<std::pair<dealii::VectorTools::internal::VectorDoFTuple<3> const, std::pair<dealii::Tensor<1, 3, double>, dealii::DoFHandler<3, 3>::cell_iterator> > > >&, std::map<dealii::VectorTools::internal::VectorDoFTuple<3>, dealii::Vector<double>, std::less<dealii::VectorTools::internal::VectorDoFTuple<3> >, std::allocator<std::pair<dealii::VectorTools::internal::VectorDoFTuple<3> const, dealii::Vector<double> > > >&)
#2  /home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/deal.II-v9.5.2/lib/libdeal_II.so.9.5.2: void dealii::VectorTools::internal::compute_nonzero_normal_flux_constraints_active_or_level<3, 3>(dealii::DoFHandler<3, 3> const&, unsigned int, std::set<unsigned int, std::less<unsigned int>, std::allocator<unsigned int> > const&, std::map<unsigned int, dealii::Function<3, double> const*, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, dealii::Function<3, double> const*> > > const&, dealii::AffineConstraints<double>&, dealii::Mapping<3, 3> const&, dealii::IndexSet const&, unsigned int)
#3  /home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/deal.II-v9.5.2/lib/libdeal_II.so.9.5.2: void dealii::VectorTools::compute_no_normal_flux_constraints<3, 3>(dealii::DoFHandler<3, 3> const&, unsigned int, std::set<unsigned int, std::less<unsigned int>, std::allocator<unsigned int> > const&, dealii::AffineConstraints<double>&, dealii::Mapping<3, 3> const&)
#4  /home/bbpanneg/software/aspect/build__initial_conditions_continental_rift/aspect-release: aspect::MeshDeformation::MeshDeformationHandler<3>::make_initial_constraints()
#5  /home/bbpanneg/software/aspect/build__initial_conditions_continental_rift/aspect-release: aspect::MeshDeformation::MeshDeformationHandler<3>::setup_dofs()
#6  /home/bbpanneg/software/aspect/build__initial_conditions_continental_rift/aspect-release: aspect::Simulator<3>::setup_dofs()
#7  /home/bbpanneg/software/aspect/build__initial_conditions_continental_rift/aspect-release: aspect::Simulator<3>::run()
#8  /home/bbpanneg/software/aspect/build__initial_conditions_continental_rift/aspect-release: void run_simulator<3>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, bool)
#9  /home/bbpanneg/software/aspect/build__initial_conditions_continental_rift/aspect-release: main
--------------------------------------------------------

Aborting!

The error can be reproduced by taking the cookbook crustal_deformation/crustal_model_3D.prm and changing the Geometry model settings from

    set X extent  = 128e3
    set Y extent  =  96e3
    set Z extent  =  16e3
    set X repetitions  = 8 
    set Y repetitions  = 3

to

    set X extent = 125e3
    set Y extent = 100e3
    set Z extent = 25e3
    set X repetitions = 5 
    set Y repetitions = 4

To get rid of the error, one only has to change the X extent to 128e3. It seems like a very specific edge case where the normal vector to the surface cannot be computed. For example, 

    set X extent  = 128e3
    set Y extent  =  96e3
    set Z extent  =  16e3
    set X repetitions  = 8 
    set Y repetitions  = 6

does work, even though it also has repetitions that lead to square surface elements.

The error occurs for any mesh deformation plugin (as it occurs in make_initial_constraints in the mesh_deformation interface) and we also tried several different setups. For a more complex one, the same error occurs but during make_constraints instead of make_initial_constraints in the mesh deformation interface.

anne-glerum commented 1 month ago

Per our discussion yesterday in the biweekly user meeting, here more info on the compiler and optimization settings:

ASPECT:

#        ASPECT_VERSION:            2.6.0-pre
#        DEAL_II_DIR:               /home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/deal.II-v9.5.2/lib/cmake/deal.II
#        DEAL_II VERSION:           9.5.2
#        ASPECT_USE_FP_EXCEPTIONS:  ON
#        ASPECT_RUN_ALL_TESTS:      OFF
#        ASPECT_USE_SHARED_LIBS:    ON
#        ASPECT_HAVE_LINK_H:        ON
#        ASPECT_WITH_LIBDAP:        OFF
#        ASPECT_WITH_NETCDF:        OFF
#        ASPECT_WITH_WORLD_BUILDER: ON /home/bbpanneg/software/aspect/aspect_initial_conditions_continental_rift/aspect/contrib/world_builder
#        ASPECT_PRECOMPILE_HEADERS: ON
#        ASPECT_UNITY_BUILD:        ON
#        CMAKE_CXX_COMPILER:        GNU 9.2.0 on platform Linux x86_64
#                                   /sw/compiler/gcc/9.2.0/skl/bin/c++
#          COMPILE_OPTIONS:         $<$<COMPILE_LANGUAGE:CXX>:-pedantic;-Wall;-Wextra;-Wmissing-braces;-Woverloaded-virtual;-Wpointer-arith;-Wsign-compare;-Wsuggest-override;-Wswitch;-Wsynth;-Wwrite-strings;-Wno-placement-new;-Wno-deprecated-declarations;-Wno-literal-suffix;-Wno-psabi;-Wno-class-memaccess;-Wno-parentheses;-Wno-unused-local-typedefs;-fopenmp-simd;-pthread;-march=native;-O2;-funroll-loops;-funroll-all-loops;-fstrict-aliasing;-Wno-unused-local-typedefs;-O3;-fdiagnostics-color=always>
#          COMPILE_DEFINITIONS:     NDEBUG
#          COMPILE_FLAGS:           
#          LINK_LIBRARIES:          dealii::dealii_release;/usr/lib64/libz.so;/home/projects/bbp00039/coupling/fastscape/fastscape_climate/build_release_20240306/libfastscapelib_fortran.so;WorldBuilderRelease;dl

dealII:

#        CMAKE_CXX_COMPILER:     GNU 9.2.0 on platform Linux x86_64
#                                /sw/compiler/gcc/9.2.0/skl/bin/c++
#  Exported compiler and linker flags:
#        DEAL_II_CXX_FLAGS:            -fopenmp-simd -pthread -march=native
#        DEAL_II_CXX_FLAGS_RELEASE:    -O2 -funroll-loops -funroll-all-loops -fstrict-aliasing -Wno-unused-local-typedefs -O3
#        DEAL_II_CXX_FLAGS_DEBUG:      -O0 -ggdb -Wa,--compress-debug-sections
#        DEAL_II_WARNING_FLAGS:        -pedantic -Wall -Wextra -Wmissing-braces -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wsuggest-override -Wswitch -Wsynth -Wwrite-strings -Wno-placement-new -Wno-deprecated-declarations -Wno-literal-suffix -Wno-psabi -Wno-class-memaccess -Wno-parentheses -Wno-unused-local-typedefs
#        DEAL_II_LINKER_FLAGS:         -rdynamic -fuse-ld=gold -pthread
#        DEAL_II_LINKER_FLAGS_RELEASE: 
#        DEAL_II_LINKER_FLAGS_DEBUG:   -ggdb -Wl,--compress-debug-sections=zlib
#        TYPE: INTERFACE_LIBRARY
#        INTERFACE_LINK_LIBRARIES: $<$<CONFIG:Debug>:dealii::dealii_debug>,$<$<CONFIG:Release>:dealii::dealii_release>
#        INTERFACE_COMPILE_OPTIONS: $<$<COMPILE_LANGUAGE:CXX>:-fopenmp-simd,-pthread,-march=native>,$<$<AND:$<CONFIG:Debug>,$<COMPILE_LANGUAGE:CXX>>:-O0,-ggdb,-Wa,--compress-debug-sections>,$<$<AND:$<CONFIG:Release>,$<COMPILE_LANGUAGE:CXX>>:-O2,-funroll-loops,-funroll-all-loops,-fstrict-aliasing,-Wno-unused-local-typedefs,-O3>
#        INTERFACE_LINK_OPTIONS: -rdynamic,-fuse-ld=gold,-pthread,$<$<CONFIG:Debug>:-ggdb,-Wl,--compress-debug-sections=zlib>,$<$<CONFIG:Release>:>
#    Target: dealii_release
#        TYPE: SHARED_LIBRARY
#        VERSION: 9.5.2
#        SOVERSION: 9.5.2
#        LINK_LIBRARIES: dl,rt,interface_lapack,interface_mpi,interface_zlib,interface_trilinos,interface_kokkos,interface_hdf5,interface_sundials,interface_p4est_release,bundled_boost_serialization_release,bundled_boost_system_release,bundled_boost_iostreams_release,bundled_tbb_release,bundled_umfpack_L_UMF_release,bundled_umfpack_Z_UMF_release,bundled_umfpack_L_UMFPACK_release,bundled_umfpack_DL_TSOLVE_release,bundled_umfpack_DL_TRIPLET_MAP_NOX_release,bundled_umfpack_DL_TRIPLET_MAP_X_release,bundled_umfpack_DL_TRIPLET_NOMAP_X_release,bundled_umfpack_DL_TRIPLET_NOMAP_NOX_release,bundled_umfpack_DL_STORE_release,bundled_umfpack_DL_ASSEMBLE_release,bundled_umfpack_DL_SOLVE_release,bundled_umfpack_ZL_TSOLVE_release,bundled_umfpack_ZL_TRIPLET_MAP_NOX_release,bundled_umfpack_ZL_TRIPLET_MAP_X_release,bundled_umfpack_ZL_TRIPLET_NOMAP_X_release,bundled_umfpack_ZL_TRIPLET_NOMAP_NOX_release,bundled_umfpack_ZL_STORE_release,bundled_umfpack_ZL_ASSEMBLE_release,bundled_umfpack_ZL_SOLVE_release,bundled_umfpack_GENERIC_release,bundled_amd_int_release,bundled_amd_long_release,bundled_amd_global_release,bundled_muparser_release,object_numerics_release,object_fe_release,object_matrix_free_release,object_dofs_release,object_lac_release,object_base_release,object_cgal_release,object_gmsh_release,object_grid_release,object_hp_release,object_multigrid_release,object_distributed_release,object_algorithms_release,object_integrators_release,object_meshworker_release,object_opencascade_release,object_particle_release,object_differentiation_ad_release,object_differentiation_sd_release,object_physics_elasticity_release,object_physics_release,object_rol_release,object_non_matching_release,object_sundials_release,object_trilinos_release,object_arborx_release
#        INCLUDE_DIRECTORIES: /home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/build/deal.II-v9.5.2/include,/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/include,/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/bundled/tbb-2018_U2/include,/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/bundled/boost-1.70.0/include,/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/bundled/umfpack/UMFPACK/Include,/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/bundled/umfpack/AMD/Include,/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/bundled/muparser_v2_3_3//include
#        COMPILE_DEFINITIONS: NDEBUG
#        COMPILE_OPTIONS: -pedantic,-Wall,-Wextra,-Wmissing-braces,-Woverloaded-virtual,-Wpointer-arith,-Wsign-compare,-Wsuggest-override,-Wswitch,-Wsynth,-Wwrite-strings,-Wno-placement-new,-Wno-deprecated-declarations,-Wno-literal-suffix,-Wno-psabi,-Wno-class-memaccess,-Wno-parentheses,-Wno-unused-local-typedefs,-fopenmp-simd,-pthread,-march=native,-O2,-funroll-loops,-funroll-all-loops,-fstrict-aliasing,-Wno-unused-local-typedefs,-O3
#        LINK_OPTIONS: -rdynamic,-fuse-ld=gold,-pthread
#        INTERFACE_LINK_LIBRARIES: dl,rt,interface_lapack,interface_mpi,interface_zlib,interface_trilinos,interface_kokkos,interface_hdf5,interface_sundials,interface_p4est_release
#        INTERFACE_INCLUDE_DIRECTORIES: $<BUILD_INTERFACE:/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/build/deal.II-v9.5.2/include>,$<BUILD_INTERFACE:/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/include>,$<INSTALL_INTERFACE:include>,$<BUILD_INTERFACE:/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/bundled/tbb-2018_U2/include>,$<BUILD_INTERFACE:/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/bundled/boost-1.70.0/include>,$<BUILD_INTERFACE:/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/bundled/umfpack/UMFPACK/Include>,$<BUILD_INTERFACE:/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/bundled/umfpack/AMD/Include>,$<BUILD_INTERFACE:/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/bundled/muparser_v2_3_3//include>,$<INSTALL_INTERFACE:include/deal.II/bundled>
#        INTERFACE_SYSTEM_INCLUDE_DIRECTORIES: $<BUILD_INTERFACE:/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/build/deal.II-v9.5.2/include>,$<BUILD_INTERFACE:/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/include>,$<INSTALL_INTERFACE:include>,$<BUILD_INTERFACE:/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/bundled/tbb-2018_U2/include>,$<BUILD_INTERFACE:/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/bundled/boost-1.70.0/include>,$<BUILD_INTERFACE:/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/bundled/umfpack/UMFPACK/Include>,$<BUILD_INTERFACE:/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/bundled/umfpack/AMD/Include>,$<BUILD_INTERFACE:/home/projects/bbp00039/dealii/dealii_v9.5.2/candi_built/tmp/unpack/deal.II-v9.5.2/bundled/muparser_v2_3_3//include>,$<INSTALL_INTERFACE:include/deal.II/bundled>
#        INTERFACE_COMPILE_DEFINITIONS: NDEBUG
anne-glerum commented 1 month ago

I also tried, but couldn't, reproduce the issue on my Mac. My Mac installation has the following details: ASPECT:

#        ASPECT_VERSION:            2.6.0-pre
#        GIT REVISION:              39e799da3 (fix_stresses_elasticity)
#        DEAL_II_DIR:               /Applications/deal.II.app/Contents/Resources/Libraries/lib/cmake/deal.II
#        DEAL_II VERSION:           9.5.0
#        ASPECT_USE_FP_EXCEPTIONS:  OFF
#        ASPECT_RUN_ALL_TESTS:      ON
#        ASPECT_USE_SHARED_LIBS:    ON
#        ASPECT_HAVE_LINK_H:        OFF
#        ASPECT_WITH_LIBDAP:        OFF
#        ASPECT_WITH_NETCDF:        OFF
#        ASPECT_WITH_WORLD_BUILDER: ON /Applications/ASPECT/VisualStudioCode/aspect/contrib/world_builder
#        ASPECT_PRECOMPILE_HEADERS: ON
#        ASPECT_UNITY_BUILD:        ON
#        CMAKE_CXX_COMPILER:        AppleClang 14.0.0.14000029 on platform Darwin x86_64
#                                   /Applications/deal.II.app/Contents/Resources/spack/opt/mpich-4.1.1-hbtt/bin/mpic++
#        DEAL_II options:
#          _WITH_CXX14:             ON
#          _WITH_CXX17:             TRUE
#          _MPI_VERSION:            
#          _WITH_64BIT_INDICES:     OFF

dealII

#        CMAKE_INSTALL_PREFIX:   /Applications/deal.II.app/Contents/Resources/spack/opt/dealii-9.5.0-z2dj
#        CMAKE_SOURCE_DIR:       /var/folders/8z/hlb6vc015qjggytkxn84m6_c0000gn/T/heltai/spack-stage/spack-stage-dealii-9.5.0-z2dj7h3yellxliquaj6yhbeeouydnohw/spack-src
#                                (version 9.5.0)
#        CMAKE_BINARY_DIR:       /private/var/folders/8z/hlb6vc015qjggytkxn84m6_c0000gn/T/heltai/spack-stage/spack-stage-dealii-9.5.0-z2dj7h3yellxliquaj6yhbeeouydnohw/spack-build-z2dj7h3
#        CMAKE_CXX_COMPILER:     AppleClang 14.0.3.14030022 on platform Darwin x86_64
#                                /Applications/deal.II.app/Contents/Resources/spack/opt/mpich-4.1.1-hbtt/bin/mpic++
#        CMAKE_C_COMPILER:       /Applications/deal.II.app/Contents/Resources/spack/opt/mpich-4.1.1-hbtt/bin/mpicc
#        CMAKE_Fortran_COMPILER: /Applications/deal.II.app/Contents/Resources/spack/opt/mpich-4.1.1-hbtt/bin/mpif90
#        CMAKE_GENERATOR:        Ninja
#        C++ language standard:  C++17
#        Vectorization level:    128 bit (sse2)
#
#  Exported compiler and linker flags:
#        DEAL_II_CXX_FLAGS:            -openmp-simd -ftemplate-depth=1024 -std=c++17
#        DEAL_II_CXX_FLAGS_RELEASE:    -O2 -funroll-loops -fstrict-aliasing -Wno-unused-local-typedefs
#        DEAL_II_CXX_FLAGS_DEBUG:      -ffp-exception-behavior=strict -Og -ggdb -Wa,--compress-debug-sections
#        DEAL_II_WARNING_FLAGS:        -pedantic -Wall -Wextra -Wmissing-braces -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wsuggest-override -Wswitch -Wsynth -Wwrite-strings -Wno-deprecated-declarations -Wno-psabi -Wfloat-conversion -Qunused-arguments -Wno-unsupported-friend -Wno-pass-failed -Wno-unused-local-typedefs
#        DEAL_II_LINKER_FLAGS:         
#        DEAL_II_LINKER_FLAGS_RELEASE: 
#        DEAL_II_LINKER_FLAGS_DEBUG:   -ggdb
#    Target: dealii
#        TYPE: INTERFACE_LIBRARY
#        INTERFACE_LINK_LIBRARIES: $<$<CONFIG:Debug>:dealii::dealii_debug>,$<$<CONFIG:Release>:dealii::dealii_release>
#        INTERFACE_COMPILE_OPTIONS: $<$<COMPILE_LANGUAGE:CXX>:-openmp-simd,-ftemplate-depth=1024,-std=c++17>,$<$<AND:$<CONFIG:Debug>,$<COMPILE_LANGUAGE:CXX>>:-ffp-exception-behavior=strict,-Og,-ggdb,-Wa,--compress-debug-sections>,$<$<AND:$<CONFIG:Release>,$<COMPILE_LANGUAGE:CXX>>:-O2,-funroll-loops,-fstrict-aliasing,-Wno-unused-local-typedefs>
#        INTERFACE_LINK_OPTIONS: $<$<CONFIG:Debug>:-ggdb>,$<$<CONFIG:Release>:>
#    Target: dealii_release
#        TYPE: SHARED_LIBRARY
#        VERSION: 9.5.0
#        SOVERSION: 9.5.0
#        LINK_LIBRARIES: interface_lapack,interface_mpi,interface_zlib,interface_boost,interface_ginkgo,interface_metis,interface_petsc,interface_trilinos,interface_umfpack,interface_kokkos,interface_adolc,interface_arborx,interface_arpack,interface_assimp,interface_cgal,interface_gmsh,interface_gsl,interface_hdf5,interface_muparser,interface_opencascade,interface_p4est,interface_scalapack,interface_slepc,interface_sundials,interface_symengine,interface_tbb_release,object_numerics_release,object_fe_release,object_matrix_free_release,object_dofs_release,object_lac_release,object_base_release,object_cgal_release,object_gmsh_release,object_grid_release,object_hp_release,object_multigrid_release,object_distributed_release,object_algorithms_release,object_integrators_release,object_meshworker_release,object_opencascade_release,object_particle_release,object_differentiation_ad_release,object_differentiation_sd_release,object_physics_elasticity_release,object_physics_release,object_rol_release,object_non_matching_release,object_sundials_release,object_trilinos_release,object_arborx_release
#        INCLUDE_DIRECTORIES: /private/var/folders/8z/hlb6vc015qjggytkxn84m6_c0000gn/T/heltai/spack-stage/spack-stage-dealii-9.5.0-z2dj7h3yellxliquaj6yhbeeouydnohw/spack-build-z2dj7h3/include,/var/folders/8z/hlb6vc015qjggytkxn84m6_c0000gn/T/heltai/spack-stage/spack-stage-dealii-9.5.0-z2dj7h3yellxliquaj6yhbeeouydnohw/spack-src/include
#        COMPILE_DEFINITIONS: NDEBUG
#        COMPILE_OPTIONS: -pedantic,-Wall,-Wextra,-Wmissing-braces,-Woverloaded-virtual,-Wpointer-arith,-Wsign-compare,-Wsuggest-override,-Wswitch,-Wsynth,-Wwrite-strings,-Wno-deprecated-declarations,-Wno-psabi,-Wfloat-conversion,-Qunused-arguments,-Wno-unsupported-friend,-Wno-pass-failed,-Wno-unused-local-typedefs,-openmp-simd,-ftemplate-depth=1024,-std=c++17,-O2,-funroll-loops,-fstrict-aliasing,-Wno-unused-local-typedefs
#        INTERFACE_LINK_LIBRARIES: interface_lapack,interface_mpi,interface_zlib,interface_boost,interface_ginkgo,interface_metis,interface_petsc,interface_trilinos,interface_umfpack,interface_kokkos,interface_adolc,interface_arborx,interface_arpack,interface_assimp,interface_cgal,interface_gmsh,interface_gsl,interface_hdf5,interface_muparser,interface_opencascade,interface_p4est,interface_scalapack,interface_slepc,interface_sundials,interface_symengine,interface_tbb_release
#        INTERFACE_INCLUDE_DIRECTORIES: $<BUILD_INTERFACE:/private/var/folders/8z/hlb6vc015qjggytkxn84m6_c0000gn/T/heltai/spack-stage/spack-stage-dealii-9.5.0-z2dj7h3yellxliquaj6yhbeeouydnohw/spack-build-z2dj7h3/include>,$<BUILD_INTERFACE:/var/folders/8z/hlb6vc015qjggytkxn84m6_c0000gn/T/heltai/spack-stage/spack-stage-dealii-9.5.0-z2dj7h3yellxliquaj6yhbeeouydnohw/spack-src/include>,$<INSTALL_INTERFACE:include>
#        INTERFACE_SYSTEM_INCLUDE_DIRECTORIES: $<BUILD_INTERFACE:/private/var/folders/8z/hlb6vc015qjggytkxn84m6_c0000gn/T/heltai/spack-stage/spack-stage-dealii-9.5.0-z2dj7h3yellxliquaj6yhbeeouydnohw/spack-build-z2dj7h3/include>,$<BUILD_INTERFACE:/var/folders/8z/hlb6vc015qjggytkxn84m6_c0000gn/T/heltai/spack-stage/spack-stage-dealii-9.5.0-z2dj7h3yellxliquaj6yhbeeouydnohw/spack-src/include>,$<INSTALL_INTERFACE:include>
#        INTERFACE_COMPILE_DEFINITIONS: NDEBUG
anne-glerum commented 1 month ago

I'll try to tease out for what coordinates and what cell the error is triggered on the cluster.

anne-glerum commented 3 weeks ago

The debugger does not allow to set breakpoints or look into the code, I guess because dealii was not built in dev mode (and I have no permissions to do so).

A more recent installation of dealii using a newer gcc alleviates the problem though:

###
#
#  ASPECT configuration:
#        ASPECT_VERSION:            2.6.0-pre
#        GIT REVISION:              e8f3a93 (main)
#        CMAKE_BUILD_TYPE:          DebugRelease
#
#        DEAL_II_DIR:               /scratch/usr/bbpanneg/software/dealii/CentOS7_candi_9.5.2/deal.II-v9.6.0/lib/cmake/deal.II
#        DEAL_II VERSION:           9.6.0
#        ASPECT_USE_FP_EXCEPTIONS:  ON
#        ASPECT_RUN_ALL_TESTS:      OFF
#        ASPECT_USE_SHARED_LIBS:    ON
#        ASPECT_HAVE_LINK_H:        ON
#        ASPECT_WITH_LIBDAP:        OFF
#        ASPECT_WITH_NETCDF:        ON
#        ASPECT_WITH_WORLD_BUILDER: ON /home/bbpanneg/software/aspect/aspect_main/aspect/contrib/world_builder
#        ASPECT_PRECOMPILE_HEADERS: ON
#        ASPECT_UNITY_BUILD:        ON
#
#        CMAKE_INSTALL_PREFIX:      /usr/local
#        CMAKE_SOURCE_DIR:          /home/bbpanneg/software/aspect/aspect_main/aspect
#        CMAKE_BINARY_DIR:          /home/bbpanneg/software/aspect/build_CentOS7__main
#        CMAKE_CXX_COMPILER:        GNU 9.3.0 on platform Linux x86_64
#                                   /sw/comm/openmpi/4.1.4/skl/gcc/bin/mpicxx
#        PARAMETER_GUI_EXECUTABLE:  PARAMETER_GUI_EXECUTABLE-NOTFOUND
#
#        LINKAGE:                   DYNAMIC
#
#        DEAL_II options:
#          _WITH_CXX14:             ON
#          _WITH_CXX17:             ON
#          _MPI_VERSION:            3.1
#          _WITH_64BIT_INDICES:     OFF
#
###