search

ECP-copa / ExaMPM

Material point method proxy application based on Cabana.
BSD 3-Clause "New" or "Revised" License
9 stars 11 forks source link

Dam break example does not generate output files #48

Closed dineshadepu closed 1 year ago

dineshadepu commented 1 year ago

Hi,

My aim is to write a SPH-DEM solver and simulate millions of particles using Cabana, and in the process I am learning to understand the internals and basics of the API.

I am successful at installing the ExaMPM software, however, the DamBreak example does not dump any output files while running.

Here is the output:

/DamBreak 0.05 2 0 0.001 1.0 50 serial
Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set
  In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads
  For best performance with OpenMP 3.1 set OMP_PROC_BIND=true
  For unit testing set OMP_PROC_BIND=false

Time 0.000000 / 1.000000
Time 0.050000 / 1.000000
...

Can someone please help with this issue?

Many thanks.

streeve commented 1 year ago

Sorry for the confusion - see the PR I just opened where it prints a message letting you know that external libraries are needed in Cabana in order to get particle output. Let me know if you have further questions

dineshadepu commented 1 year ago

Thank you for your prompt response. I have added the flags to the cmake argument. I running into trouble getting parallel hdf5 linked while compiling Cabana. I have very little experience with c++ build systems and compilation procedures. I will put forward what I have worked on towards this issue.

>>>  adepudinesh@UoSDellLaptop (master) /home/adepudinesh/Kokkos/Cabana/build $  
|  DELL LAPTOP => cmake       -D CMAKE_BUILD_TYPE="Debug"       -D CMAKE_PREFIX_PATH=$KOKKOS_INSTALL_DIR       -D CMAKE_INSTALL_PREFIX=$CABANA_INSTALL_DIR       -D Cabana_REQUIRE_OPENMP=ON       -D Cabana_ENABLE_EXAMPLES=ON   -D Cabana_ENABLE_PERFORMANCE_TESTING=OFF       -D Cabana_ENABLE_CAJITA=ON  -D Cabana_REQUIRE_SILO=OFF  -D Cabana_REQUIRE_HDF5=ON  .. ;

-- Enabled Kokkos devices: OPENMP;SERIAL
-- HDF5 C compiler wrapper is unable to compile a minimal HDF5 program.
-- Cabana Revision = 'd24d188b0c4a33118ac96ddd81d7c6462f921b8a'
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- Configuring done
-- Generating done
-- Build files have been written to: /home/adepudinesh/Kokkos/Cabana/build

From the seconds line, it says that HDF5 C compiler wrapper is unable to compile a minimal HDF5 program.. I am not sure, why I am getting this error. I have both libhdf5-serial-dev and libhdf5-mpi-dev installed. I went ahead and tried make install, which resulted in the following error,

 adepudinesh@UoSDellLaptop (master) /home/adepudinesh/Kokkos/Cabana/build $  
|  DELL LAPTOP UoS => make install
Consolidate compiler generated dependencies of target HelloWorld
[  1%] Building CXX object example/core_tutorial/01_hello_world/CMakeFiles/HelloWorld.dir/hello_world.cpp.o
In file included from /home/adepudinesh/Kokkos/Cabana/core/src/Cabana_Core.hpp:53,
                 from /home/adepudinesh/Kokkos/Cabana/example/core_tutorial/01_hello_world/hello_world.cpp:12:
/home/adepudinesh/Kokkos/Cabana/core/src/Cabana_HDF5ParticleOutput.hpp: In function ‘void Cabana::Experimental::HDF5ParticleOutput::Impl::writeFields(Cabana::Experimental::HDF5ParticleOutput::HDF5Config, hid_t, std::size_t, std::size_t, hsize_t, int, const char*, const char*, const SliceType&, typename std::enable_if<(2 == typename SliceType::kokkos_view::traits::dimension::rank), int*>::type)’:
/home/adepudinesh/Kokkos/Cabana/core/src/Cabana_HDF5ParticleOutput.hpp:275:9: error: there are no arguments to ‘H5Pset_dxpl_mpio’ that depend on a template parameter, so a declaration of ‘H5Pset_dxpl_mpio’ must be available [-fpermissive]
  275 |         H5Pset_dxpl_mpio( plist_id, H5FD_MPIO_COLLECTIVE );
      |         ^~~~~~~~~~~~~~~~
/home/adepudinesh/Kokkos/Cabana/core/src/Cabana_HDF5ParticleOutput.hpp:275:9: note: (if you use ‘-fpermissive’, G++ will accept your code, but allowing the use of an undeclared name is deprecated)
/home/adepudinesh/Kokkos/Cabana/core/src/Cabana_HDF5ParticleOutput.hpp: In function ‘void Cabana::Experimental::HDF5ParticleOutput::Impl::writeFields(Cabana::Experimental::HDF5ParticleOutput::HDF5Config, hid_t, std::size_t, std::size_t, hsize_t, int, const char*, const char*, const SliceType&, typename std::enable_if<(3 == typename SliceType::kokkos_view::traits::dimension::rank), int*>::type)’:
/home/adepudinesh/Kokkos/Cabana/core/src/Cabana_HDF5ParticleOutput.hpp:352:9: error: there are no arguments to ‘H5Pset_dxpl_mpio’ that depend on a template parameter, so a declaration of ‘H5Pset_dxpl_mpio’ must be available [-fpermissive]
  352 |         H5Pset_dxpl_mpio( plist_id, H5FD_MPIO_COLLECTIVE );
      |         ^~~~~~~~~~~~~~~~
/home/adepudinesh/Kokkos/Cabana/core/src/Cabana_HDF5ParticleOutput.hpp: In function ‘void Cabana::Experimental::HDF5ParticleOutput::Impl::writeFields(Cabana::Experimental::HDF5ParticleOutput::HDF5Config, hid_t, std::size_t, std::size_t, hsize_t, int, const char*, const char*, const SliceType&, typename std::enable_if<(4 == typename SliceType::kokkos_view::traits::dimension::rank), int*>::type)’:
/home/adepudinesh/Kokkos/Cabana/core/src/Cabana_HDF5ParticleOutput.hpp:433:9: error: there are no arguments to ‘H5Pset_dxpl_mpio’ that depend on a template parameter, so a declaration of ‘H5Pset_dxpl_mpio’ must be available [-fpermissive]
  433 |         H5Pset_dxpl_mpio( plist_id, H5FD_MPIO_COLLECTIVE );
      |         ^~~~~~~~~~~~~~~~
/home/adepudinesh/Kokkos/Cabana/core/src/Cabana_HDF5ParticleOutput.hpp: In function ‘void Cabana::Experimental::HDF5ParticleOutput::writeTimeStep(Cabana::Experimental::HDF5ParticleOutput::HDF5Config, const string&, MPI_Comm, int, double, std::size_t, const CoordSliceType&, FieldSliceTypes&& ...)’:
/home/adepudinesh/Kokkos/Cabana/core/src/Cabana_HDF5ParticleOutput.hpp:522:5: error: there are no arguments to ‘H5Pset_fapl_mpio’ that depend on a template parameter, so a declaration of ‘H5Pset_fapl_mpio’ must be available [-fpermissive]
  522 |     H5Pset_fapl_mpio( plist_id, comm, MPI_INFO_NULL );
      |     ^~~~~~~~~~~~~~~~
/home/adepudinesh/Kokkos/Cabana/core/src/Cabana_HDF5ParticleOutput.hpp:535:9: error: there are no arguments to ‘H5Pset_all_coll_metadata_ops’ that depend on a template parameter, so a declaration of ‘H5Pset_all_coll_metadata_ops’ must be available [-fpermissive]
  535 |         H5Pset_all_coll_metadata_ops( plist_id, 1 );
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/adepudinesh/Kokkos/Cabana/core/src/Cabana_HDF5ParticleOutput.hpp:536:9: error: there are no arguments to ‘H5Pset_coll_metadata_write’ that depend on a template parameter, so a declaration of ‘H5Pset_coll_metadata_write’ must be available [-fpermissive]
  536 |         H5Pset_coll_metadata_write( plist_id, 1 );
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/adepudinesh/Kokkos/Cabana/core/src/Cabana_HDF5ParticleOutput.hpp:607:9: error: there are no arguments to ‘H5Pset_dxpl_mpio’ that depend on a template parameter, so a declaration of ‘H5Pset_dxpl_mpio’ must be available [-fpermissive]
  607 |         H5Pset_dxpl_mpio( plist_id, H5FD_MPIO_COLLECTIVE );
      |         ^~~~~~~~~~~~~~~~
/home/adepudinesh/Kokkos/Cabana/core/src/Cabana_HDF5ParticleOutput.hpp: In function ‘void Cabana::Experimental::HDF5ParticleOutput::readTimeStep(Cabana::Experimental::HDF5ParticleOutput::HDF5Config, const string&, MPI_Comm, int, std::size_t, const string&, double&, FieldSliceType&)’:
/home/adepudinesh/Kokkos/Cabana/core/src/Cabana_HDF5ParticleOutput.hpp:760:5: error: there are no arguments to ‘H5Pset_fapl_mpio’ that depend on a template parameter, so a declaration of ‘H5Pset_fapl_mpio’ must be available [-fpermissive]
  760 |     H5Pset_fapl_mpio( plist_id, comm, MPI_INFO_NULL );
      |     ^~~~~~~~~~~~~~~~
/home/adepudinesh/Kokkos/Cabana/core/src/Cabana_HDF5ParticleOutput.hpp:765:9: error: there are no arguments to ‘H5Pset_all_coll_metadata_ops’ that depend on a template parameter, so a declaration of ‘H5Pset_all_coll_metadata_ops’ must be available [-fpermissive]
  765 |         H5Pset_all_coll_metadata_ops( plist_id, 1 );
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/adepudinesh/Kokkos/Cabana/core/src/Cabana_HDF5ParticleOutput.hpp:818:9: error: there are no arguments to ‘H5Pset_dxpl_mpio’ that depend on a template parameter, so a declaration of ‘H5Pset_dxpl_mpio’ must be available [-fpermissive]
  818 |         H5Pset_dxpl_mpio( plist_id, H5FD_MPIO_COLLECTIVE );
      |         ^~~~~~~~~~~~~~~~
make[2]: *** [example/core_tutorial/01_hello_world/CMakeFiles/HelloWorld.dir/build.make:76: example/core_tutorial/01_hello_world/CMakeFiles/HelloWorld.dir/hello_world.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:722: example/core_tutorial/01_hello_world/CMakeFiles/HelloWorld.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

I have looked this up, and it says I need parallel HDF5 installed. Which I have done using the instructions given https://github.com/HDFGroup/hdf5/blob/develop/release_docs/INSTALL . I have installed parallel hdf5 library locally in downloads folder, as below

 ----UoS LAPTOP DELL LAPTOP DELL LAPTOP ----------------------------------
 adepudinesh@UoSDellLaptop /home/adepudinesh/Downloads/hdf5-1.14.1-2/bin $  
|  DELL LAPTOP UoS => ls
batch         config.guess   h5redeploy     Makefile.am       README.md
bbrelease     config.sub     h5redeploy.in  Makefile.in       release
buildhdf5     debug-ohdr     h5vers         makehelp          restore.sh
checkapi      depcomp        install-sh     make_overflow     runbkgprog
checkposix    format_source  iostats        make_vers         switch_maint_mode
chkcopyright  genparser      ltmain.sh      missing           test-driver
cmakehdf5     h5cc           make_err       output_filter.sh  trace
compile       h5cc.in        Makefile       pkgscrpts         warnhist
(base)  

 ----UoS LAPTOP DELL LAPTOP DELL LAPTOP ----------------------------------
 adepudinesh@UoSDellLaptop /home/adepudinesh/Downloads/hdf5-1.14.1-2/bin $  
|  DELL LAPTOP UoS => h5cc --showconfig
gcc -L/home/adepudinesh/Downloads/hdf5-1.14.1-2/hdf5/lib /home/adepudinesh/Downloads/hdf5-1.14.1-2/hdf5/lib/libhdf5_hl.a /home/adepudinesh/Downloads/hdf5-1.14.1-2/hdf5/lib/libhdf5.a -lz -ldl -lm -Wl,-rpath -Wl,/home/adepudinesh/Downloads/hdf5-1.14.1-2/hdf5/lib -I/usr/lib/x86_64-linux-gnu/openmpi/include -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi
(base)

Further I have added it to my path

# for parallel hdf5 for Cabana
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/adepudinesh/Downloads/hdf5-1.14.1-2/src/.libs
export PATH=/home/adepudinesh/Downloads/hdf5-1.14.1-2/bin:$PATH
export PATH=/home/adepudinesh/Downloads/hdf5-1.14.1-2/src:$PATH
export PATH=/home/adepudinesh/Downloads/hdf5-1.14.1-2/src/.libs:$PATH
export HDF5_ROOT=/home/adepudinesh/Downloads/hdf5-1.14.1-2

Even after this, I still get the same error. I am not sure how to proceed from here. Is this the right approach?

Thank you for your help. :)

dineshadepu commented 1 year ago

I have tried with Silo input output format. I have installed it at /home/adepudinesh/Kokkos/Silo and while running Cabana cmake command, I used the following command

cmake       -D CMAKE_BUILD_TYPE="Debug"       
-D CMAKE_PREFIX_PATH=$KOKKOS_INSTALL_DIR       
-D CMAKE_INSTALL_PREFIX=$CABANA_INSTALL_DIR       
-D Cabana_REQUIRE_OPENMP=ON       
-D Cabana_ENABLE_EXAMPLES=ON       
-D Cabana_ENABLE_TESTING=OFF       
-D Cabana_ENABLE_PERFORMANCE_TESTING=OFF       
-D Cabana_ENABLE_CAJITA=ON  
-D Cabana_REQUIRE_SILO=ON  
-D Cabana_REQUIRE_HDF5=OFF 
-D CMAKE_PREFIX_PATH=/home/adepudinesh/Kokkos/Silo/include 
-D CMAKE_PREFIX_PATH=/home/adepudinesh/Kokkos/Silo/lib  .. ;

I figured out that I need to add the paths of the Silo installation to cmake command through, CMAKE_PREFIX_PATH argument. This way I could compile the Cabana library with output functionality.

I will look into hdf5 later, and update.

Thank you.

streeve commented 1 year ago

You mentioned you have both serial and parallel HDF5 installed - if that is the case you will need to directly point to the parallel install location (Cabana still finds the serial version first otherwise). And you can generally pass the top level install location, not the separate include/library directories (although you can set those separately if you really want to): -D CMAKE_PREFIX_PATH="$KOKKOS_INSTALL_DIR;$HDF5_INSTALL_DIR"

dineshadepu commented 1 year ago

Thank you for the instruction. Yes, I have installed them though apt-get, as well as I have built the parallel version from sources. I do not understand much of c++ yet. I will try hdf5 after a few weeks.

dineshadepu commented 1 year ago

I was able to successfully install Cabana with parallel HDF5. However, while I am running exampm, I am running into this cmake error

 adepudinesh@UoSDellLaptop (master) /home/adepudinesh/post_doc/softwares/ExaMPM/build $  
|  DELL LAPTOP UoS => cmake -D CMAKE_PREFIX_PATH="/home/adepudinesh/post_doc/softwares/kokkos/build/install;/home/adepudinesh/post_doc/softwares/Cabana/build/install" ..
-- The CXX compiler identification is GNU 11.3.0
-- The C compiler identification is GNU 11.3.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found OpenMP_C: -fopenmp (found version "4.5") 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Enabled Kokkos devices: OPENMP;SERIAL
-- Found MPI_CXX: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1") found components: CXX 
-- Found MPI_C: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1")  
-- Found SILO: /home/adepudinesh/post_doc/softwares/Silo/SiloInstall/include  
CMake Error at /usr/share/cmake-3.22/Modules/CMakeFindDependencyMacro.cmake:47 (find_package):
  find_package called with invalid argument "1.14.1-2"
Call Stack (most recent call first):
  /home/adepudinesh/post_doc/softwares/Cabana/build/install/lib/cmake/Cabana/CabanaConfig.cmake:47 (find_dependency)
  CMakeLists.txt:11 (find_package)

-- Found CLANG_FORMAT: /usr/bin/clang-format (found suitable version "14.0.0", minimum required is "14") 
-- Configuring incomplete, errors occurred!
See also "/home/adepudinesh/post_doc/softwares/ExaMPM/build/CMakeFiles/CMakeOutput.log".

I have install hdf5 with hdf5-1.14.1-2 source files. The line at CabanaConfig.cmake:47 is

find_dependency(HDF5 1.14.1-2 REQUIRED COMPONENTS C)

My hdf5 installation is at 

/home/adepudinesh/Downloads/hdf5/install/

Can you please help me with fixing this cmake error?

dineshadepu commented 1 year ago

I have manually removed the extra 1.14.1-2 from find_dependency command, and tried to run cmake again. This fixed my problem.