[DRAFT for testing] Support IntelLLVM compiler

DusanJovic-NOAA commented 1 month ago

Commit Queue Requirements:

[ ] Fill out all sections of this template.
[ ] All sub component pull requests have been reviewed by their code managers.
[ ] Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
[ ] Commit 'test_changes.list' from previous step

Description:

Add preliminary support for Intel LLVM compiler. Added new top-level cmake/IntelLLVM.cmake configuration file. Update CMakeLists.txt in various submodules to recognize IntelLLVM as supported compiler and add Intel LLVM compiler flags.

Useful porting guide: https://www.intel.com/content/www/us/en/developer/articles/guide/porting-guide-for-ifort-to-ifx.html

At this moment I only tried to build the model on Hercules.

In order to use new compiler you need to set the following three environment variables:

export I_MPI_CC=icx export I_MPI_CXX=icpx export I_MPI_F90=ifx

Commit Message:

* UFSWM - 
  * AQM - 
  * CDEPS - 
  * CICE - 
  * CMEPS - 
  * CMakeModules - 
  * FV3 - 
    * ccpp-physics - 
    * atmos_cubed_sphere - 
  * GOCART - 
  * HYCOM - 
  * MOM6 - 
  * NOAHMP - 
  * WW3 - 
  * stochastic_physics -

Priority:

Normal

Git Tracking

UFSWM:

Closes #2025

Sub component Pull Requests:

AQM:
CDEPS:
CICE:
CMEPS:
CMakeModules:
FV3:
- ccpp-physics:
- atmos_cubed_sphere:
GOCART:
HYCOM:
MOM6:
NOAHMP:
WW3:
stochastic_physics:
None

UFSWM Blocking Dependencies:

Blocked by #
None

Changes

Regression Test Changes (Please commit test_changes.list):

PR Adds New Tests/Baselines.
PR Updates/Changes Baselines.
No Baseline Changes.

Input data Changes:

None.
New input data.
Updated input data.

Library Changes/Upgrades:

Required
- Library names w/versions:
- Git Stack Issue (JCSDA/spack-stack#)
No Updates

Testing Log:

RDHPCS
- [ ] Hera
- [ ] Orion
- [ ] Hercules
- [ ] Jet
- [ ] Gaea
- [ ] Derecho
WCOSS2
- [ ] Dogwood/Cactus
- [ ] Acorn
[ ] CI
[ ] opnReqTest (complete task if unnecessary)

BrianCurtis-NOAA commented 1 month ago

Is this something for only Hercules, or will other systems be implementing IntelLLVM ?

DusanJovic-NOAA commented 1 month ago

Is this something for only Hercules, or will other systems be implementing IntelLLVM ?

Eventually we must switch to IntelLLVM on all platforms. When, I don't know. But classic Intel compilers are now deprecated, see: https://www.intel.com/content/www/us/en/developer/articles/release-notes/oneapi-fortran-compiler-release-notes.html

NOTE: Intel® Fortran Compiler Classic (ifort) is now deprecated and will be discontinued in late 2024. Intel recommends that customers transition now to using the LLVM-based Intel® Fortran Compiler (ifx) for continued Windows and Linux support, new language support, new language features, and optimizations. For more information on ifx, see the Intel® Fortran Compiler Developer Guide and Reference and the Porting Guide for ifort Users to ifx.

DusanJovic-NOAA commented 1 month ago

Compilation with Intel LLVM on Hera fails with this error:

/scratch2/NCEPDEV/fv3-cam/Dusan.Jovic/ufs/support_intelllvm/ufs-weather-model/MOM6-interface/
MOM6/src/framework/MOM_unit_testing.F90(128): error #5533: Feature found on this line is not 
yet supported in ifx                                                                         
  test%proc => proc
--^
compilation aborted for /scratch2/NCEPDEV/fv3-cam/Dusan.Jovic/ufs/support_intelllvm/ufs-weath
er-model/MOM6-interface/MOM6/src/framework/MOM_unit_testing.F90 (code 3)
make[2]: *** [MOM6-interface/CMakeFiles/mom6_obj.dir/build.make:1765: MOM6-interface/CMakeFil
es/mom6_obj.dir/MOM6/src/framework/MOM_unit_testing.F90.o] Error 3

DusanJovic-NOAA commented 1 month ago

All coupled debug tests fail on Hercules with this error:

179: forrtl: severe (408): fort: (3): Subscript #3 of the array UVEL has value 0 which is less than the lower bound of 1
179: 
179: Image              PC                Routine            Line        Source             
179: ufs_model          000000000B5A1262  accum_hist               2318  ice_history.F90
179: ufs_model          000000000B4BFB6F  initializerealize         835  ice_comp_nuopc.F90
179: ufs_model          0000000000B94024  Unknown               Unknown  Unknown
179: ufs_model          0000000000B97F9F  Unknown               Unknown  Unknown
179: ufs_model          0000000000D0EBC7  Unknown               Unknown  Unknown
179: ufs_model          0000000000CFB022  Unknown               Unknown  Unknown

This is looks like a bug in the ifx.

DeniseWorthen commented 1 month ago

@NickSzapiro-NOAA Would you please take a look at what is happening here w/ CICE? You'll need to get from @DusanJovic-NOAA build instructions and maybe a sandbox run directory.

NickSzapiro-NOAA commented 1 month ago

Of course, @DeniseWorthen @DusanJovic-NOAA. Please let me know if there are any special instructions needed to reproduce. On first pass, it's curious that the ice speed may be the first call to accum_hist with an array that was not allocated: https://github.com/NOAA-EMC/CICE/blob/emc/develop/cicecore/cicedyn/analysis/ice_history.F90#L2317-L2320

DusanJovic-NOAA commented 1 month ago

I was able to create a small test program that reproduces an error identical to the one we see in CICE. Dom (@climbfuji) confirmed that it works with the latest Intel LLVM compiler (ifx (IFX) 2024.1.0 20240308), so it is probably just a bug in the older versions.

Version on Hercules is (ifx (IFX) 2023.1.0 20230320). On Hera it's even older (ifx (IFORT) 2022.0.0 20211123), and that version can not even compile the code with MOM6. See above error.

We need to find a machine (NOAA R&D) on which we can get the latest version of Intel LLVM compiler (2024.1.0) installed.

climbfuji commented 1 month ago

I can install it on Hercules in the EPIC spack-stack space, but we still need to manage building all dependencies with it.

junwang-noaa commented 1 month ago

If it is confirmed that the latest Intel LLVM compiler fixes the issue, let's ask sysadmin to install the latest version.

DusanJovic-NOAA commented 1 month ago

I can install it on Hercules in the EPIC spack-stack space, but we still need to manage building all dependencies with it.

Eventually yes, but for now I think we can still build all libraries (spack-stack) with ifort (but the new version 2024) and only the model with ifx. Object files and module files should be binary compatible between ifort and ifx.

According to Intel: ... Binaries and libraries generated with ifort can be linked with binaries and libraries built with ifx, and .mod files generated with one compiler can be used by the other (64-bit targets only). Both compilers use the the same runtime libraries. ...

climbfuji commented 1 month ago

Except that there's no more icc anc icpc, so at least for those we have to switch to icx and icpx. It should be fairly straightforward to build just the dependencies for the ufs-weather-model with ifort and/or ifx and get all the other packages compiled in a second step.

NickSzapiro-NOAA commented 1 month ago

@DusanJovic-NOAA Does filling worka first (https://github.com/NOAA-EMC/CICE/blob/emc/develop/cicecore/cicedyn/analysis/ice_history.F90#L2321-L2331) give a temporary fix?

DusanJovic-NOAA commented 1 month ago

@DusanJovic-NOAA Does filling worka first (https://github.com/NOAA-EMC/CICE/blob/emc/develop/cicecore/cicedyn/analysis/ice_history.F90#L2321-L2331) give a temporary fix?

I think it should. Looks like having an array expression (that depends on iblk) as an actual subroutine argument is what fails.

ufs-community / ufs-weather-model