Exawind / amr-wind

AMReX-based structured wind solver
https://exawind.github.io/amr-wind
Other
103 stars 78 forks source link

Segfault of AMR-Wind + OpenFAST with oneapi #1048

Closed marchdf closed 1 month ago

marchdf commented 1 month ago

A little over a month ago, I was told that people were having segfaults using amr-wind + openfast on Kestrel and using oneapi compilers. I was able to reproduce and get a stack trace: Screenshot 2024-04-05 at 2 29 30 PM

It is a fairly annoying bug because it does not show up with gcc and (at the time) the oneapi compilers were not able to build in debug. I tried to reproduce earlier this week but now, the oneapi compilers won't build OpenFAST 3.5.3:

[ 79%] Building Fortran object modules/aerodyn/CMakeFiles/aerodynlib.dir/src/AeroDyn_Inflow.f90.o
cd /kfs2/projects/hfm/mhenryde/exawind/exawind-manager/stage/spack-stage-openfast-3.5.3-rnpmeveceruzfwsmgf46fhj3ibgf4vps/spack-build-rnpmeve/modules/aerodyn && /opt/cray/pe/mpich/8.1.28/ofi/intel/2022.1/bin/mpif90 -DHAS_FORTRAN2008_FEATURES -DOPENFAST_DOUBLE_PRECISION -Daerodynlib_EXPORTS -I/kfs2/projects/hfm/mhenryde/exawind/exawind-manager/stage/spack-stage-openfast-3.5.3-rnpmeveceruzfwsmgf46fhj3ibgf4vps/spack-build-rnp\
meve/ftnmods -fpic -fpp -real-size 64 -double-size 64 -fiopenmp -O3 -module ../../ftnmods -fPIC -c /kfs2/projects/hfm/mhenryde/exawind/exawind-manager/environments/amr-wind-of-intel/openfast/modules/aerodyn/src/AeroDyn_Inflow.f90 -o CMakeFiles/aerodynlib.dir/src/AeroDyn_Inflow.f90.o
 #0 0x0000000002228e97 (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x2228e97)
 #1 0x0000000002228e66 (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x2228e66)
 #2 0x000000000212b121 (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x212b121)
 #3 0x000000000212b356 (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x212b356)
 #4 0x000000000212bab7 (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x212bab7)
 #5 0x000000000212a70b (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x212a70b)
 #6 0x0000000002192890 (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x2192890)
 #7 0x00000000021acd3b (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x21acd3b)
 #8 0x00000000021f9737 (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x21f9737)
 #9 0x00000000021f8473 (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x21f8473)
#10 0x00000000021f9027 (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x21f9027)
#11 0x0000000002279b4d (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x2279b4d)
#12 0x0000000002277705 (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x2277705)
#13 0x0000000002277ce4 (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x2277ce4)
#14 0x00000000022752b2 (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x22752b2)
#15 0x0000000002277705 (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x2277705)
#16 0x000000000227495a (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x227495a)
#17 0x0000000002277705 (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x2277705)
#18 0x0000000002111c05 (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x2111c05)
#19 0x00000000021115bd (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x21115bd)
#20 0x00000000022e13ce (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x22e13ce)
#21 0x00007f025ea04d85 __libc_start_main (/lib64/libc.so.6+0x3ad85)
#22 0x0000000001f51aa9 (/nopt/nrel/apps/cpu_stack/compilers/02-24/spack/opt/spack/linux-rhel8-sapphirerapids/gcc-12.2.1/intel-oneapi-compilers-2023.2.0-hwdq5hei2obxznfjhtlav4mi5h5jd4zw/compiler/2023.2.0/linux/bin-llvm/xfortcom+0x1f51aa9)

/kfs2/projects/hfm/mhenryde/exawind/exawind-manager/environments/amr-wind-of-intel/openfast/modules/aerodyn/src/AeroDyn_Inflow.f90: error #5623: **Internal compiler error: internal abort** Please report this error along with the circumstances in which it occurred in a Software Problem Report.  Note: File and line given may not be explicit cause of this error.
# 1 "/kfs2/projects/hfm/mhenryde/exawind/exawind-manager/environments/amr-wind-of-intel/openfast/modules/aerodyn/src/AeroDyn_Inflow.f90"
^
compilation aborted for /kfs2/projects/hfm/mhenryde/exawind/exawind-manager/environments/amr-wind-of-intel/openfast/modules/aerodyn/src/AeroDyn_Inflow.f90 (code 3)
make[2]: *** [modules/aerodyn/CMakeFiles/aerodynlib.dir/build.make:429: modules/aerodyn/CMakeFiles/aerodynlib.dir/src/AeroDyn_Inflow.f90.o] Error 3
make[2]: Leaving directory '/kfs2/projects/hfm/mhenryde/exawind/exawind-manager/stage/spack-stage-openfast-3.5.3-rnpmeveceruzfwsmgf46fhj3ibgf4vps/spack-build-rnpmeve'
make[1]: *** [CMakeFiles/Makefile2:725: modules/aerodyn/CMakeFiles/aerodynlib.dir/all] Error 2
make[1]: Leaving directory '/kfs2/projects/hfm/mhenryde/exawind/exawind-manager/stage/spack-stage-openfast-3.5.3-rnpmeveceruzfwsmgf46fhj3ibgf4vps/spack-build-rnpmeve'
make: *** [Makefile:139: all] Error 2

The current "fix" AFAIK is to just use gcc. I will be working on this next week hopefully.

jrood-nrel commented 1 month ago

The segfault with the Intel compiler itself has to do with OpenMP being enabled by default and AeroDyn_Inflow.f90 segfaults the compiler.

marchdf commented 1 month ago

I tried this case again on kestrel and made sure openmp was for real disabled in openfast:

modified   CMakeLists.txt
@@ -110,7 +110,7 @@ include(${CMAKE_SOURCE_DIR}/cmake/set_rpath.cmake)
 # OpenMP
 #-------------------------------------------------------------------------------

-if (OPENMP OR BUILD_FASTFARM OR BUILD_OPENFAST_CPP_API)
+if (OPENMP)
   if (OPENMP)
     FIND_PACKAGE(OpenMP REQUIRED)
   else()

(exawind-manager should do this automatically for us). And the case now works and doesn't segfault as noted above. Closing this issue.