search

TACC / Galaxy

Galaxy is an asynchronous parallel visualization ray tracer for performant rendering in distributed computing environments. Galaxy builds upon Intel OSPRay and Intel Embree, including ray queueing and sending logic inspired by TACC GraviT.
Apache License 2.0
19 stars 9 forks source link

stand up MPM on Frontera #17

Open pnav opened 3 years ago

pnav commented 3 years ago

Get MPM code running on Frontera as a basis for in situ integration

kks32 commented 3 years ago

I have ran CB-Geo MPM on Frontera. The instructions on how to get MPM running for LoneStar5 works for Frontera too (https://mpm.cb-geo.com/#/user/run/hpc/tacc-ls5). Except for the VTK libraries. VTK libraries are still not available as a module on Frontera. The workaround for now is to do:

setenv VTK_LOCATION=/work/01197/semeraro/frontera/VTK/VTKBinary

andrewsolis commented 3 years ago

thanks, I was wondering to do about the vtk workaround. I'll continue taking a look.

andrewsolis commented 3 years ago

It also looks like eigen is no longer maintained on bitbucket and is now maintained inside gitlab. https://gitlab.com/libeigen/eigen

pnav commented 3 years ago

I have ran CB-Geo MPM on Frontera. The instructions on how to get MPM running for LoneStar5 works for Frontera too (https://mpm.cb-geo.com/#/user/run/hpc/tacc-ls5). Except for the VTK libraries. VTK libraries are still not available as a module on Frontera. The workaround for now is to do:

setenv VTK_LOCATION=/work/01197/semeraro/frontera/VTK/VTKBinary

Thanks Krishna! Regrettably it is not happy with my build of KaHIP.. any ideas? I notice your KaHIP is from Feb 2020, perhaps there's been some incompatible drift? It appears MPM wants three KaHIP sub-specs now, rather than just a root.

CC=icc CXX=icpc cmake -DBOOST_ROOT=$TACC_BOOST_DIR -DBOOST_INCLUDE_DIRS=$TACC_BOOST_INC -DCMAKE_BUILD_TYPE=Release -DEIGEN3_INCLUDE_DIR=$TACC_EIGEN_INC -DKAHIP_LIBRARIES=/scratch1/00401/pnav/dev/KaHIP/build/libinterface.so -DKAHIP_INCLUDE_DIRS=/scratch1/00401/pnav/dev/KaHIP/build -DKAHIP_TEST_RUNS=/scratch1/00401/pnav/dev/KaHIP/build/interface_test -DVTK_LOCATION=$TACC_VTK_DIR ..
-- The CXX compiler identification is Intel 19.1.0.20200306
-- Check for working CXX compiler: /opt/intel/compilers_and_libraries_2020.1.217/linux/bin/intel64/icpc
-- Check for working CXX compiler: /opt/intel/compilers_and_libraries_2020.1.217/linux/bin/intel64/icpc -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Boost: /opt/apps/intel19/python3_7/boost/1.72/lib/cmake/Boost-1.72.0/BoostConfig.cmake (found version "1.72.0") found components: filesystem system 
-- Found MPI_CXX: /opt/intel/compilers_and_libraries_2020.1.217/linux/mpi/intel64/lib/libmpicxx.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1")  
-- The C compiler identification is Intel 19.1.0.20200306
-- Check for working C compiler: /opt/intel/compilers_and_libraries_2020.1.217/linux/bin/intel64/icc
-- Check for working C compiler: /opt/intel/compilers_and_libraries_2020.1.217/linux/bin/intel64/icc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- HDF5: Using hdf5 compiler wrapper to determine CXX configuration
-- Found HDF5: /usr/lib64/libhdf5_cpp.so;/usr/lib64/libhdf5.so;/usr/lib64/libsz.so;/usr/lib64/libz.so;/usr/lib64/libdl.so;/usr/lib64/libm.so (found version "1.8.12") found components:  CXX HL 
-- Found OpenMP_C: -qopenmp (found version "5.0") 
-- Found OpenMP_CXX: -qopenmp (found version "5.0") 
-- Found OpenMP: TRUE (found version "5.0")  
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Found MKL: /opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/include  
-- Checking for package 'KaHIP'
CMake Error at cmake/FindPackageHandleStandardArgs.cmake:137 (message):
  KaHIP could not be found/configured.  (missing: KAHIP_LIBRARIES)
Call Stack (most recent call first):
  cmake/FindPackageHandleStandardArgs.cmake:377 (_FPHSA_FAILURE_MESSAGE)
  cmake/FindKaHIP.cmake:140 (find_package_handle_standard_args)
  CMakeLists.txt:106 (find_package)

-- Configuring incomplete, errors occurred!
See also "/scratch1/00401/pnav/dev/mpm/build/CMakeFiles/CMakeOutput.log".
semeraro commented 3 years ago

The situation with VTK is this. VTKpython only works with version 3.8 or higher of python3 (it works fine with 2.7). We have python3.8.2 on frontera, however, it requires gcc9.1 (which we have). I have a gcc9.1/python3.8.2 version of vtk built. It is installed in ~semeraro/rpminstall. One can access it via module use ~semeraro/rpminstall and load it like a module. Alternatively there exists /work/01197/semeraro/VTK/VTKBinary38 and VTKBinary27 which contain vtk built against python3.8 and python2.7 respectively. I have hesitated to install vtk in a system location due to the whacky dependencies for python3. Im happy to build a non python version in a system location if you folks need it. All this is on frontera

pnav commented 3 years ago

thanks @andrewsolis for pointing out that there's a magic build script in the KaHIP project. That worked! MPM successfully configured.

andrewsolis commented 3 years ago

@kks32 I was able to configure and compile mpm, but I'm not sure if it is built with VTK. When I try and change the VTK_DIR option in ccmake to either the VTKBinary27 or VTKBinary38 directory it tries to re-configure but then shows that for the variable VTK_DIR it is not found, yet I can generate and call make to build mpm. Does this mean that it is built without VTK? I assume VTk is optional for mpm but for this usecase it is necessary.

UPDATE : I was able to build with VTK successfully, thanks @semeraro .

kks32 commented 3 years ago

I have added the steps to compile mpm with VTK on Frontera: https://mpm.cb-geo.com/#/user/run/hpc/tacc-frontera

Here's the single script to do the same.

module load boost hdf5
cd $WORK && git clone https://gitlab.com/libeigen/eigen.git
cd $WORK && git clone https://github.com/KaHIP/KaHIP && \
   cd KaHIP && sh ./compile_withcmake.sh 
git clone https://github.com/cb-geo/mpm-benchmarks.git
git clone https://github.com/cb-geo/mpm.git && cd mpm
export CC=icc
export CXX=icpc
mkdir build && cd build &&  cmake -DBOOST_INCLUDE_DIRS=$TACC_BOOST_INC -DCMAKE_BUILD_TYPE=Release -DEIGEN3_INCLUDE_DIR=$WORK/eigen -DKAHIP_ROOT=$WORK/KaHIP -DVTK_DIR=/work/01197/semeraro/frontera/VTK/VTKBinary/ ..
make -j

@pnav We can now close this issue?

kks32 commented 3 years ago

I found SWR was not properly loaded before, which lead to failure at linker level. Here is the updated set of instructions: https://mpm.cb-geo.com/#/user/run/hpc/tacc-frontera

module load boost hdf5 swr/20.0.5
export LD_LIBRARY_PATH=$TACC_SWR_LIB:$LD_LIBRARY_PATH
cd $WORK && git clone https://gitlab.com/libeigen/eigen.git
cd $WORK && git clone https://github.com/KaHIP/KaHIP && \
   cd KaHIP && sh ./compile_withcmake.sh 
cd $WORK
git clone https://github.com/cb-geo/mpm.git
git clone https://github.com/cb-geo/mpm-benchmarks.git benchmarks
export CC=icc
export CXX=icpc
cd mpm && mkdir build && cd build &&  cmake -DBOOST_INCLUDE_DIRS=$TACC_BOOST_INC -DCMAKE_BUILD_TYPE=Release -DEIGEN3_INCLUDE_DIR=$WORK/eigen -DKAHIP_ROOT=$WORK/KaHIP -DVTK_ROOT=/work/01197/semeraro/frontera/VTK/VTKBinary/ ..
make -j