search

JCSDA / spack-stack

Creative Commons Zero v1.0 Universal
27 stars 48 forks source link

Update ECMWF libraries to latest versions #1340

Open climbfuji opened 1 month ago

climbfuji commented 1 month ago

Is your feature request related to a problem? Please describe. JCSDA would like to update the ECMWF libraries eckit, fckit, ectrans, and atlas to their latest versions. We first need to test and determine which versions to move to, but the goal is to use the latest of each. Here are the latest tags as of 11/22/24.

Describe the solution you'd like Here is a check list for JCSDA to fill in with the known good versions to upgrade to. This needs to be verified and filled out first, then we can make the updates in spack, spack-stack to move to these versions

Checklist

Track here the PRs that move the configuration to the specified versions above.

srherbener commented 2 weeks ago

@climbfuji, I have confirmed that JCSDA does want ectrans@1.5.0 to be included in spack-stack-1.9.0.

srherbener commented 2 weeks ago

I can start by adding the new version to the ectrans package.py formula in the authoritative repo.

srherbener commented 2 weeks ago

I think it will be more useful to make this issue about the ECMWF library set (which includes ectrans) that JCSDA is interested in upgrading, since we have to select a set of libraries that are compatible with each other. I have modified this issue to track this.

fmahebert commented 1 week ago

I have tested the following tags as a set:

The JEDI bundle builds with these tags, and nearly all expected ctests pass (*). Aside from some questions around odc (that I'm not an expert on; see notes below), I think we can proceed with these tags.

Notes

srherbener commented 1 week ago

@fmahebert thanks for testing the latest versions of the ecmwf libraries! I think the results are very encouraging.

spack-stack-1.8.0 and the spack-stack develop branch both are using odc 1.5.2 which is the latest version. The spack-stack formula for odc restricts optimization levels to 1 in the release related build types (Release, RelWithDebInfo). Perhaps that is the difference.

I think it's pretty safe to target the versions you tested so I will update the checklist.

fmahebert commented 1 week ago

spack-stack-1.8.0 and the spack-stack develop branch both are using odc 1.5.2 which is the latest version. The spack-stack formula for odc restricts optimization levels to 1 in the release related build types (Release, RelWithDebInfo). Perhaps that is the difference.

Yes, this is different from what I did. Do you know if it's easy to set odc-specific build options like this when building odc from the jedi-bundle? If I can easily test this with a more correctly-configured odc, I will do so.

srherbener commented 1 week ago

I think you could edit the odc CMakeLists.txt file and add:

set (ENABLE_FORTRAN OFF)
set (CMAKE_C_FLAGS_RELWITHDEBINFO "-O1")
set (CMAKE_CXX_FLAGS_RELWITHDEBINFO "-O1")

This is what the spack formula for ODC is doing (via -D command line options). I just noticed that the two REWITHDEBINFO settings are only applied for macos (apple-clang version 14.0.3 specifically), so on an HPC platform only the ENABLE_FORTRAN setting is used.

fmahebert commented 1 week ago

@srherbener I tested again with odc 1.5.2 configured for ENABLE_FORTRAN OFF, and this didn't change anything. However, I realized also that the failing ioda ctests were trying to call the odc executable in the ioda_compare.sh script, and not finding the executable. Turns out my in-bundle setup didn't have the build/bin dir in the path, thus the odc exec could not be found. After updating my PATH variable, the ioda ctests pass.

Therefore, as far as I can tell, the tags listed above are working for JEDI.

fmahebert commented 1 week ago

And ECMWF has just tagged atlas version 0.40.0; I will run my tests again with this tag and report back here.

fmahebert commented 1 week ago

I tested again with atlas 0.40.0, and this works just as well as 0.39.0 did — most jedi-bundle ctests pass on GNU and on intel (and the failures exist also with spack-stack 1.7 version of ecmwf repos).

So I would recommend updating to this set of tags:

@srherbener would it be ok to bump the atlas version to the latest 0.40.0?

srherbener commented 1 week ago

@fmahebert absolutely fine to bump atlas to 0.40.0. I'll update the description. Thank you very much for all the work to test these versions! I think we have a solid plan now.