search

jsitaraman / tioga

Tioga is a library for overset grid assembly on parallel distributed systems
GNU Lesser General Public License v3.0
64 stars 36 forks source link

Build issue with Spack #54

Closed psakievich closed 3 years ago

psakievich commented 3 years ago

I'm trying to build tioga for nalu-wind. However, I'm getting the following error when I try to build with gcc 7:

     129    [ 72%] Building Fortran object driver/CMakeFiles/tioga.exe.dir/testTioga.F90.o
     130    cd /tmp/psakiev/spack-stage/spack-stage-tioga-develop-2654ulebty7znslynmhhia2wmyxz3p45/spack-build-2654ule/driver && /pro
            jects/wind/spack-manager/spack/lib/spack/env/gcc/gfortran -D-pthread -DTIOGA_HAS_NODEGID -I/sierra/sntools/SDK/mpi/openmp
            i/4.0.5-gcc-7.2.0-RHEL7/include -I/projects/sierra/linux_rh7/SDK/mpi/openmpi/4.0.5-gcc-7.2.0-RHEL7/include -I/sierra/snto
            ols/SDK/mpi/openmpi/4.0.5-gcc-7.2.0-RHEL7/lib -I/tmp/psakiev/spack-stage/spack-stage-tioga-develop-2654ulebty7znslynmhhia
            2wmyxz3p45/spack-src/src -fbounds-check -fbacktrace -fdefault-real-8 -ffree-line-length-none -O3 -DNDEBUG -O3 -fPIE -cpp 
            -c /tmp/psakiev/spack-stage/spack-stage-tioga-develop-2654ulebty7znslynmhhia2wmyxz3p45/spack-src/driver/testTioga.F90 -o 
            CMakeFiles/tioga.exe.dir/testTioga.F90.o
     131    <command-line>:1:0:
     132    
  >> 133    Error: macro names must be identifiers
  >> 134    make[2]: *** [driver/CMakeFiles/tioga.exe.dir/testTioga.F90.o] Error 1
     135    make[2]: Leaving directory `/tmp/psakiev/spack-stage/spack-stage-tioga-develop-2654ulebty7znslynmhhia2wmyxz3p45/spack-bui
            ld-2654ule'
  >> 136    make[1]: *** [driver/CMakeFiles/tioga.exe.dir/all] Error 2
     137    make[1]: Leaving directory `/tmp/psakiev/spack-stage/spack-stage-tioga-develop-2654ulebty7znslynmhhia2wmyxz3p45/spack-bui
            ld-2654ule'
  >> 138    make: *** [all] Error 2

pretty sure the issue is the flag -D-pthread being fed to the fortran compiler. I can't seem to track down where this flag is being created. So far I've only seen this on rhel7 and gcc7. I don't see this issue when I build on mac-os with gcc 8., but the -D-pthread flag is missing in that build. Those are the only two complier combinations I've tested this on so far.

sayerhs commented 3 years ago

@psakievich is this a CUDA/HIP build, or pure CPU build. @aprokop had noticed a similar issue in #47 and the solution there was to set the Fortran compiler to mpif90

psakievich commented 3 years ago

interesting. This is just pure cpu right now. Gonna try GPU soon so I will update on how that does if I don't find the solution. I'm trying to chase down through the CMakeFiles right now to see if I can find where this gets injected.

sayerhs commented 3 years ago

This is not something injected by Tioga CMake, it is coming from a combination of MPI and PThread find package rules.

svdavidson commented 3 years ago

@psakievich, I had the same problem building tioga (exawind branch) for nalu-wind using gcc 8.4.0 on ppc64le. However, tioga on the master branch builds successfully. So maybe there is a hint there.

psakievich commented 3 years ago

Thanks @svdavidson. I'm not sure what the differences are between the two branches. I'll have to look at them closer. I'm just starting to look at tioga for the first time. I ran into another build error with ifort this morning. If I revert the last commit on the exawind branch it builds fine. I haven't had a chance to look too closely at what is causing this yet. 

3 errors found in build log:
     133    [ 72%] Building Fortran object driver/CMakeFiles/poisson_mms.exe.dir/poisson_mms.f90.o
     134    cd /projects/wind/spack-manager/spack/var/spack/stage/psakiev/spack-stage-tioga-develop-ihz7xycwe5psq2mao5z2pjrfu2cjnhss/spack-build-ihz7xyc/driver && /projects/wind/spack-manager/sp
            ack/lib/spack/env/intel/ifort -DTIOGA_HAS_NODEGID -I/opt/openmpi/4.0/intel/include -I/opt/openmpi/4.0/intel/lib -I/projects/wind/spack-manager/spack/var/spack/stage/psakiev/spack-sta
            ge-tioga-develop-ihz7xycwe5psq2mao5z2pjrfu2cjnhss/spack-src/src -r8 -double_size 128 -O3 -fPIE -c /projects/wind/spack-manager/spack/var/spack/stage/psakiev/spack-stage-tioga-develop
            -ihz7xycwe5psq2mao5z2pjrfu2cjnhss/spack-src/driver/poisson_mms.f90 -o CMakeFiles/poisson_mms.exe.dir/poisson_mms.f90.o
     135    [ 74%] Linking CXX executable poisson_mms.exe
     136    cd /projects/wind/spack-manager/spack/var/spack/stage/psakiev/spack-stage-tioga-develop-ihz7xycwe5psq2mao5z2pjrfu2cjnhss/spack-build-ihz7xyc/driver && /projects/wind/spack-manager/sp
            ack/opt/spack/linux-rhel7-sandybridge/intel-19.0.3.199/cmake-3.20.3-uno43e5pjqawx232xmxscgo5mpjervrw/bin/cmake -E cmake_link_script CMakeFiles/poisson_mms.exe.dir/link.txt --verbose=
            1
     137    /projects/wind/spack-manager/spack/lib/spack/env/intel/icpc -O3 -DNDEBUG CMakeFiles/poisson_mms.exe.dir/poisson_mms.f90.o -o poisson_mms.exe  -Wl,-rpath,/opt/openmpi/4.0/intel/lib:::
            ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
            ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: libtiogadriver.a ../src/libtioga.a /opt/openmpi/4.0/intel/lib/libmpi_usempif08.so /opt/ope
            nmpi/4.0/intel/lib/libmpi_usempi_ignore_tkr.so /opt/openmpi/4.0/intel/lib/libmpi_mpifh.so /opt/openmpi/4.0/intel/lib/libmpi.so -ldl -lifport -lifcoremt -lpthread
     138    /lib/../lib64/crt1.o: In function `_start':
  >> 139    (.text+0x20): undefined reference to `main'
  >> 140    make[2]: *** [driver/CMakeFiles/poisson_mms.exe.dir/build.make:105: driver/poisson_mms.exe] Error 1
     141    make[2]: Leaving directory '/projects/wind/spack-manager/spack/var/spack/stage/psakiev/spack-stage-tioga-develop-ihz7xycwe5psq2mao5z2pjrfu2cjnhss/spack-build-ihz7xyc'
  >> 142    make[1]: *** [CMakeFiles/Makefile2:172: driver/CMakeFiles/poisson_mms.exe.dir/all] Error 2
     143    make[1]: Leaving directory '/projects/wind/spack-manager/spack/var/spack/stage/psakiev/spack-stage-tioga-develop-ihz7xycwe5psq2mao5z2pjrfu2cjnhss/spack-build-ihz7xyc'
     144    make: *** [Makefile:139: all] Error 2
svdavidson commented 3 years ago

Hi @psakievich, I did some more digging and found the problem. Two of the tioga CMake files add the MPI_Fortran_COMPILE_FLAGS to the target_compile_definitions instead of target_compile_options which causes the command line option -D-pthread instead of just -pthread. I'll try creating a PR to apply the changes.

psakievich commented 3 years ago

@svdavidson thanks so much. While you're at it could you also add the -nofor-main flag. I believe that is the cause of the second compiler issue I posted here for intel. https://software.intel.com/content/www/us/en/develop/documentation/fortran-compiler-oneapi-dev-guide-and-reference/top/compiler-reference/compiler-options/compiler-option-details/linking-or-linker-options/nofor-main.html (Thanks to @jrood-nrel)

svdavidson commented 3 years ago

Sorry @psakievich, I already created pull request #55 before I saw your comment.

psakievich commented 3 years ago

@svdavidson that's okay. If you don't want to add it to #55 then I can open a separate PR. Either way works for me. That was a great CMake find BTW.

svdavidson commented 3 years ago

@psakievich, I put the #55 changes into the master branch, so the changes will need to be merged back into the exawind branch. Is that something you do on a regular basis, or should I go ahead and apply them to the exawind branch also?

psakievich commented 3 years ago

@svdavidson either way works for me. I don't regularly update tioga but I also still need to take care of the intel FORTRAN issue so I can put in a PR when I fix that and include a cherry-pick of your commit.

svdavidson commented 3 years ago

@psakievich - the spack build of nalu-wind with the exawind branch of tioga works now. I believe this issue can be closed. Thanks!