search

NCAR / spack-gust

Spack production user software stack on the Gust test system
4 stars 0 forks source link

intel (default), intel-classic, intel-oneAPI compiler modules #43

Open theurich opened 1 year ago

theurich commented 1 year ago

I would like to make the suggestion to provide three different Intel module flavors on Gust for now, with longer term only maintaining two of them. My suggestion is based on what I have seen done on similar systems, specifically NOAA's GaeaC5 R&D machine that is hosted at ORNL.

The three module flavors proposed are:

My current understanding is that Cray/Intel recommend the first combo (default) for production run codes. The "classic" option is currently available to help with the transition, but not maintained long term, b/c Intel is planning to remove the icpc+icc support later this year. Finally, intel-oneapi with ifx on the Fortran side might not be viable for all codes yet, since it still lacks some support for some language features.

Loading any of these three modules results in the Cray wrappers (ftn, CC, cc) to point to the respective intel compiler front-ends as indicated above. This is how things are set up on GaeaC5, and it does allow to use the Cray wrappers consistently. The reason I am suggesting the same setup for Gust is that I have found the three Intel module flavors on GaeaC5 very convenient. E.g. we carry out nightly ESMF regression testing with all three flavors. The module flavors make it very clear what Intel compiler combination we are testing with for each of the test suites.

Let me also add that currently on Gust, e.g. when I load oneapi/2023.0.0, I seem to be getting what is identical to the "default" flavor from above. What I mean by that is the following mapping of Cray wrappers to Intel front-ends: ftn->ifort, CC->icpx, cc->icx. It is a bit confusing to me that the "oneapi" module actually looks more like a mixture of old ifort and new icpx/icx. Just naming that module "intel", as the default might be better, sort of steering users toward this module for now. And then intel-oneapi is really using all of the oneapi front-ends.

Finally, I also understand that every system installation has its own unique requirements. I just want to share here what I have found convenient and helpful on another similar system. Thanks.

vanderwb commented 1 year ago

@theurich - we've discussed this proposal and will likely implement it in the next version of our ncarenv stack, which I anticipate deploying in March.

Eventually, we expect we will filter things down to just intel once ifx is the preferred Fortran compiler, as I suspect this is the naming scheme HPE and other Cray sites will adopt.

Thanks for the suggestion.

theurich commented 1 year ago

@vanderwb - thank you for the update. I am looking forward to the updated version of the ncarenv stack. Thank you for moving this forward.

vanderwb commented 1 year ago

@theurich - based on your experience of other systems, would you expect to see the same version number used for both compiler variants (that being the Toolkit version), or would you expect to see the actual compiler version used for the module?

For the current release we have, the versions are:

Toolkit - 2023.0.0 OneAPI - 2023.0.0 Classic - 2021.8.0

Basically, I'm asking whether other sites use intel-classic/2021.8.0 or intel-classic/2023.0.0. Thanks!

theurich commented 1 year ago

@vanderwb - that is an excellent question. Personally it makes more sense to me to version the module by the actual compiler version, rather than the toolkit. However, because of the mixed intel default currently (see below), it actually makes sense to go with the toolkit version for all the modules. And that is in fact what I am seeing done:

Specifically, I load intel-classic/2022.0.2, and I get:

--------------------------------------------------------------
Fortran Compiler version:
Intel(R) Fortran Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.5.0 Build 20211109_000000
Copyright (C) 1985-2021 Intel Corporation.  All rights reserved.
--------------------------------------------------------------
C++ Compiler version:
Intel(R) C++ Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.5.0 Build 20211109_000000
Copyright (C) 1985-2021 Intel Corporation.  All rights reserved.
--------------------------------------------------------------
C Compiler version:
Intel(R) C Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.5.0 Build 20211109_000000
Copyright (C) 1985-2021 Intel Corporation.  All rights reserved.

I.e., the version of the actual classic compiler is different from the toolkit version!

Then I load intel/2022.0.2 (which is the mixed ifort/icpx/icx default recommended right now), and I get:

--------------------------------------------------------------
Fortran Compiler version:
Intel(R) Fortran Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.5.0 Build 20211109_000000
Copyright (C) 1985-2021 Intel Corporation.  All rights reserved.
--------------------------------------------------------------
C++ Compiler version:
Intel(R) oneAPI DPC++/C++ Compiler for applications running on Intel(R) 64, Version 2022.0.0 Build 20211123
Copyright (C) 1985-2021 Intel Corporation. All rights reserved.
--------------------------------------------------------------
C Compiler version:
Intel(R) oneAPI DPC++/C++ Compiler for applications running on Intel(R) 64, Version 2022.0.0 Build 20211123
Copyright (C) 1985-2021 Intel Corporation. All rights reserved.

Finally I load intel-oneapi/2022.0.2 and I get:

--------------------------------------------------------------
Fortran Compiler version:
Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 2022.0.0 Build 20211123
Copyright (C) 1985-2021 Intel Corporation. All rights reserved.
--------------------------------------------------------------
C++ Compiler version:
Intel(R) oneAPI DPC++/C++ Compiler for applications running on Intel(R) 64, Version 2022.0.0 Build 20211123
Copyright (C) 1985-2021 Intel Corporation. All rights reserved.
--------------------------------------------------------------
C Compiler version:
Intel(R) oneAPI DPC++/C++ Compiler for applications running on Intel(R) 64, Version 2022.0.0 Build 20211123
Copyright (C) 1985-2021 Intel Corporation. All rights reserved.
vanderwb commented 1 year ago

Thanks Gerhard.

Today I've deployed a new version of the stack with your suggestions for Intel (among other changes and additions). It is not default, but if you module load ncarenv/23.03; module reset, you should be able to explore the new setup. Let me know if you run into any issues.

jedwards4b commented 1 year ago

I was able to build the parallelio esmf cesm software stack with all three of these - it works in all three cases.

theurich commented 1 year ago

@vanderwb - all three modules work as expected and test out fine with the automated ESMF regression tests. Thank you very much for providing explicit modules for all three flavors of the Intel suite. It makes it very clear.

One question: my scripts still explicitly unload module cray-libsci as per a previous recommendation. Could/should I remove this explicit unloading now, or will it not work?

jedwards4b commented 1 year ago

@theurich I don't know if this answers your question but for the spack installed esmf on gust I am using cray-libsci for all compilers except the 3 flavors of intel which use mkl.

theurich commented 1 year ago

Well, I just ran tests with all three intel flavors again with the cray-libsci module loaded (i.e. I did not explicitly unload it in the ESMF tests scripts). It still all works fine. So it looks like the original issue that was causing issues when that module was loaded has been addressed. Not saying it needs to be loaded either... just that it does not hurt now. @vanderwb - I think this issue can be closed. Thanks again!!