search

JCSDA / spack-stack

Creative Commons Zero v1.0 Universal
27 stars 47 forks source link

[INSTALL]: spack-stack with Intel compiler 2023.2.0/impi 2023.2.0 #1147

Closed junwang-noaa closed 1 month ago

junwang-noaa commented 5 months ago

Package name

spack-stack

Package version/tag

1.6.0

Build options

Intel compiler 2023.2.0 and impi 2023.2.0 on Hera.

Installation timeframe

We got some issues with SFS long time forecast, please see the issue https://github.com/ufs-community/ufs-weather-model/issues/2320 Would you please install a test version so that we can see if the issues is gone with the new Intel compiler?

Platforms

climbfuji commented 5 months ago

@junwang-noaa Please specify which HPC. Also, the most recent Intel compiler is 2024.1.2 (which has icx (no icc), icpx (no icpc), ifx and ifort).

junwang-noaa commented 5 months ago

Hera. I updated the issue. Thanks.

climbfuji commented 5 months ago

@RatkoVasic-NOAA Would you be able to do this?

RatkoVasic-NOAA commented 5 months ago

Sure, I'm currently busy with Orion installations, but I think I can squeeze this one in my schedule.

RatkoVasic-NOAA commented 5 months ago

@junwang-noaa I just checked, he was using old version of spack-stack:

To Reproduce:
The experiment used the following hash of https://github.com/ufs-community/ufs-weather-model
commit 5bec704
Author: Brian Curtis 64433609+BrianCurtis-NOAA@users.noreply.github.com
Date: Fri May 31 14:52:06 2024 -0400

I suggest he update to 1.6.0 first (newer version of ufs-weather-model is using new spack-stack), and then try other options.

junwang-noaa commented 5 months ago

@RatkoVasic-NOAA Please see Denise's testing with c48 with the latest model code with spack-stack 1.6.0. The issue is related to Intel compiler. Thanks

RatkoVasic-NOAA commented 5 months ago

@junwang-noaa , which one do you want to try with new compiler, 1.5.1, or 1.6.0?

junwang-noaa commented 5 months ago

1.6.0, I listed it in the issue description.

AlexanderRichert-NOAA commented 4 months ago

@junwang-noaa please try /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/ue-intel-2023.2.0/install/modulefiles/Core

junwang-noaa commented 4 months ago

@AlexanderRichert-NOAA Thanks for installing the library, this intel/impi version does resolve the problem!

May I ask if we can have this spack-stack library installed on all other R&D platforms?

AlexanderRichert-NOAA commented 4 months ago

It's tricky because 2023.2 (2023.2.1, specifically) is the last ever release of the classic compilers. As far as platforms, Gaea, Hercules, and Orion don't have 2023.2 (they all have 2023.1). I don't know about S4 since I don't have access.

Looping in @climbfuji. If it wouldn't adversely affect other applications, we could see about requesting admins to install 2023.2.

climbfuji commented 4 months ago

S4 has 2023.0.0.

There wouldn't be any problem moving up to 2023.2.1 from an app point of view - actually a good idea to use the very latest release that ships with the classic compilers.

The problem is that such install requests can take a looooooong time. Another option is to install the compiler ourselves under the spack-stack directory structure. That is easy, but it does take a bit of our time. If we do that, we probably want to also pick ONE RDHPCS platform were we also install 2024.2.0 to test the new LLVM compilers for C and C++ (still has the classic ifort compiler).

AlexanderRichert-NOAA commented 4 months ago

Yeah I think it's probably best to do our own installations. I'll create an issue.

climbfuji commented 4 months ago

I agree, but to limit our workload we need to be selective creating new installations for older spack-stack releases. For 1.8.0, it's totally fine to update to 2023.2.1 as the default intel compiler for all applications. For previous releases, you need to decide for which systems you want to add 1.6.0 (and/or 1.7.0) environments for these compilers.

climbfuji commented 3 months ago

According to https://github.com/JCSDA/spack-stack/issues/1170, UFS S2S needs 2023.2.1 or later and we should install Intel oneAPI 2023.2.4 (the last release of oneAPI with the classic Intel compilers) on each system.

Is this issue therefore moot?

climbfuji commented 2 months ago

I heard back about S4. The system upgrade is planned for the week of September 16, and it will include the installation of the Intel oneAPI 2023.2 suite (not sure if 2023.2.2 or 2023.2.4), and also of a GNU base environment (gcc@13.2 and openmpi). The OS will be Rocky8 (similar to Hera).

Since this is after the release of spack-stack-1.8.0, my suggestion is to remove the spack-stack-1.8.0 label from this issue, skip S4 during the initial roll out of 1.8.0, and update the site config/install spack-stack-1.8.0 with the new compilers after the OS upgrade. @srherbener Thoughts?

srherbener commented 2 months ago

For clarity's sake, are you proposing to wait until the new oneAPI compilers are available on S4 before installing spack-stack-1.8.0 on S4, OR are you proposing that we install 1.8.0 on S4 using the old compilers and retrofit the new compilers into the 1.8.0 installation when the new oneAPI compilers become available? I think the latter would be preferred since we can move to 1.8.0 earlier (using the old compilers) and be a bit more insulated from any delays that might occur with the new oneAPI compiler installation.

@eap do you agree?

climbfuji commented 2 months ago

Up to you since you are doing the installs ;-) By the time spack-stack-1.8.0 is available it's only going to be two weeks until the system is taken offline for a complete update (OS, compilers, and therefore also spack site config).

climbfuji commented 1 month ago

See https://github.com/JCSDA/spack-stack/issues/1170#issuecomment-2395781446 - since S4 was the last system that was missing in the issue description above, I can close this as completed