Closed Thomas-Ulrich closed 6 months ago
davschneller
commented
1 year ago You could try the branch https://github.com/SeisSol/PUMGen/tree/davschneller/bypass-apf . I've tried to get around using PUMI/apf (which in total seems to be a bit of a relic from the Netcdf mesh era?); however, the code may not completely be optimized for memory consumption yet. But maybe it works now already; on small meshes it does.
Thomas-Ulrich
commented
1 year ago Thank you. I cannot compile with gcc/12.2
72 /import/exception-dump/ulrich/spack/lib/spack/env/gcc/g++ -DH5_BUILT_AS_DYNAMIC_LIB -DLOG_LEVEL=2 -DPARALLEL -DUSE_HDF -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE
-D_POSIX_C_SOURCE=200809L -I/tmp/ulrich/spack-stage/spack-stage-pumgen-bypass-apf-66in2kth545lc272udok4sxexxp4laib/spack-src/src -I/tmp/ulrich/spack-stage/spack-stage-pumgen-bypass-apf-66in2kth545
lc272udok4sxexxp4laib/spack-src/submodules -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/pumi-2.2.8-mrdynb2pr2irnayun3j7g53itoxsdrjj/include -isystem /import/
exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/hdf5-1.12.2-xwxzdqakdlyy5sknts5wilegc5dlrboi/include -isystem /import/exception-dump/ulrich/myLibs/spack-packages/linux-de
bian11-zen2/gcc-12.2.0/openmpi-4.1.5-ab5rikqfa63jmfrqq54jugtlhyw3iu7d/include -O2 -std=c++17 -fopenmp -pthread -MD -MT CMakeFiles/pumgen.dir/src/aux/MPIConvenience.cpp.o -MF CMakeFiles/pumgen.dir/s
rc/aux/MPIConvenience.cpp.o.d -o CMakeFiles/pumgen.dir/src/aux/MPIConvenience.cpp.o -c /tmp/ulrich/spack-stage/spack-stage-pumgen-bypass-apf-66in2kth545lc272udok4sxexxp4laib/spack-src/src/aux/MPICo
nvenience.cpp
73 In file included from /tmp/ulrich/spack-stage/spack-stage-pumgen-bypass-apf-66in2kth545lc272udok4sxexxp4laib/spack-src/src/aux/MPIConvenience.cpp:1:
>> 74 /tmp/ulrich/spack-stage/spack-stage-pumgen-bypass-apf-66in2kth545lc272udok4sxexxp4laib/spack-src/src/aux/MPIConvenience.h:6:75: error: 'size_t' in namespace 'std' does not name a type
75 6 | void sparseAlltoallv(const void* sendbuf, const int* sendsize, const std::size_t* senddisp,
76 | ^~~~~~
>> 77 /tmp/ulrich/spack-stage/spack-stage-pumgen-bypass-apf-66in2kth545lc272udok4sxexxp4laib/spack-src/src/aux/MPIConvenience.h:8:33: error: 'size_t' in namespace 'std' does not name a type
78 8 | const std::size_t* recvdisp, MPI_Datatype recvtype, MPI_Comm comm);
79 | ^~~~~~
>> 80 make[2]: *** [CMakeFiles/pumgen.dir/build.make:205: CMakeFiles/pumgen.dir/src/aux/MPIConvenience.cpp.o] Error 1
81 make[2]: *** Waiting for unfinished jobs....
82 make[2]: Leaving directory '/tmp/ulrich/spack-stage/spack-stage-pumgen-bypass-apf-66in2kth545lc272udok4sxexxp4laib/spack-build-66in2kt'
>> 83 make[1]: *** [CMakeFiles/Makefile2:88: CMakeFiles/pumgen.dir/all] Error 2
84 make[1]: Leaving directory '/tmp/ulrich/spack-stage/spack-stage-pumgen-bypass-apf-66in2kth545lc272udok4sxexxp4laib/spack-build-66in2kt'
>> 85 make: *** [Makefile:139: all] Error 2Hi, thanks; I've updated the branch so that it includes cstddef in some more positions; the error should (hopefully) not occur anymore.
Thomas-Ulrich
commented
1 year ago The previous setup worked on 30 nodes. During the volume mesh generation the ram was around 680Gb, and during conversion, it peaked around 715 (just loosly monitoring). In short, issue fixed! Thanks again!
Tue Oct 31 11:52:39, Info: AR statistics:
Tue Oct 31 11:52:50, Info: AR max: 19.1125
Tue Oct 31 11:52:50, Info: AR (target: < ~10):
Tue Oct 31 11:52:50, Info: [ 0.00 , 2.00 ): 593689
Tue Oct 31 11:52:50, Info: [ 2.00 , 4.00 ): 728857614
Tue Oct 31 11:52:50, Info: [ 4.00 , 6.00 ): 22297227
Tue Oct 31 11:52:50, Info: [ 6.00 , 10.00 ): 2089021
Tue Oct 31 11:52:50, Info: [ 10.00 , 20.00 ): 30328
Tue Oct 31 11:52:50, Info: [ 20.00 , 40.00 ): 0
Tue Oct 31 11:52:50, Info: [ 40.00 , 100.00 ): 0
Tue Oct 31 11:52:50, Info: [ 100.00 ,inf): 0
Tue Oct 31 11:52:50, Info: Iterating over mesh to get data...
Tue Oct 31 11:52:50, Info: Counting part 0 / 1
Tue Oct 31 11:52:50, Info: Local cells: 6352716
Tue Oct 31 11:52:50, Info: Local vertices: 1112747
Tue Oct 31 11:52:50, Info: Local vertices (with duplicates): 1288241
Tue Oct 31 11:52:50, Info: Processing part 0 / 1
Tue Oct 31 11:52:50, Info: Vertices: 0 to 1112747
Tue Oct 31 11:52:55, Info: Local vertices (really, now): 258298
Tue Oct 31 11:52:55, Info: Processing part 0 / 1
Tue Oct 31 11:52:55, Info: Connectivity: 0 to 6352716
Tue Oct 31 11:52:56, Info: Groups
Tue Oct 31 11:52:57, Info: Boundaries
Tue Oct 31 11:52:57, Info: Parsed mesh successfully, writing output...
Tue Oct 31 11:53:05, Info: Total cell count: 753867879
Tue Oct 31 11:53:05, Info: Total vertex count: 128515465
Tue Oct 31 11:53:29, Info: Minimum insphere found: 4.75306
Tue Oct 31 11:53:29, Info: Writing cells
Tue Oct 31 11:54:07, Info: Writing vertices
Tue Oct 31 11:54:11, Info: Writing group information
Tue Oct 31 11:54:19, Info: Writing boundary condition
Tue Oct 31 11:54:22, Info: Writing XDMF file
Tue Oct 31 11:54:26, Info: Finished successfullyThat is great news! Then I'll try to make the PR ready some time soon-ish; it should make APF completely optional for Simmodeler and any serial mesh file (GMSH etc. should also work without APF; though I'll have to test that).
The only thing that we'd really be missing are the options described around here: https://github.com/SeisSol/PUMGen/blob/3fb1875df66c7ce14790af7d6dd3521a6ef71fcb/src/pumgen.cpp#L172-L192
I'm trying to generate a larger mesh for the texascale. The mesh (750M cells) is created by pumgen, but there is then a bug with APF. Might be a memory overflow, but we have 999Gb of RAM on the server (exception, and I was the only one using at the time).
I'm using the mesh64 branch, but this is not a new bug of this branch as I experienced it before. (pumgen compiled with spack, with:
spack install pumgen@mesh64 +with_simmetrix ^pumi@2.2.8 ^simmetrix-simmodsuite@2023.0-230923 ^easi jit=impalajit,lua(and the package modified so that it know the mesh64 version).