Aerosol-related PET log error message

[DeniseWorthen]

Description

I compiled and ran both the cpld_control_nowave_noaero_p8 and cpld_bmark_p8 in debug mode using @ulmononian's feature/add_c5 branch. Both tests run, but the bmark test (which has aerosols) produces an PET log error message on PEs 000 through 0766. There is no error message in the 0767 log. The aerosol component runs on PEs 0:767. I don't know why the last Aerosol PE does not produce the error.

The error message is:

20230907 124949.789 INFO             PET0766 UFS Aerosols: Advancing from 2013-04-01T00:00:00 to 2013-04-01T00:05:00
20230907 125011.127 ERROR            PET0766 ESMCI_Info.C:668 Info::erase() Not found  - [json.exception.out_of_range.403] key 'GridCornerLons:' not found
20230907 125011.127 ERROR            PET0766 ESMCI_Info.C:688 Info::erase() Not found  - Internal subroutine call returned Error
20230907 125011.127 ERROR            PET0766 ESMC_InfoCDef.C:243 ESMC_InfoErase() Not found  - Internal subroutine call returned Error
20230907 125011.127 ERROR            PET0766 ESMF_Info.F90:2656 ESMF_InfoRemove() Not found  - Internal subroutine call returned Error
20230907 125011.127 ERROR            PET0766 src/Superstructure/AttributeAPI/interface/ESMF_Attribute.F90:46022 ESMF_AttributeRemoveAttPackGrid( Not found  - Internal subroutine call returned Error
20230907 125011.127 ERROR            PET0766 ESMCI_Info.C:668 Info::erase() Not found  - [json.exception.out_of_range.403] key 'GridCornerLats:' not found
20230907 125011.127 ERROR            PET0766 ESMCI_Info.C:688 Info::erase() Not found  - Internal subroutine call returned Error
20230907 125011.127 ERROR            PET0766 ESMC_InfoCDef.C:243 ESMC_InfoErase() Not found  - Internal subroutine call returned Error
20230907 125011.127 ERROR            PET0766 ESMF_Info.F90:2656 ESMF_InfoRemove() Not found  - Internal subroutine call returned Error
20230907 125011.127 ERROR            PET0766 src/Superstructure/AttributeAPI/interface/ESMF_Attribute.F90:46022 ESMF_AttributeRemoveAttPackGrid( Not found  - Internal subroutine call returned Error
20230907 125011.127 ERROR            PET0766 ESMCI_Info.C:668 Info::erase() Not found  - [json.exception.out_of_range.403] key 'GridCornerLons:' not found
20230907 125011.127 ERROR            PET0766 ESMCI_Info.C:688 Info::erase() Not found  - Internal subroutine call returned Error
20230907 125011.127 ERROR            PET0766 ESMC_InfoCDef.C:243 ESMC_InfoErase() Not found  - Internal subroutine call returned Error
20230907 125011.127 ERROR            PET0766 ESMF_Info.F90:2656 ESMF_InfoRemove() Not found  - Internal subroutine call returned Error
20230907 125011.127 ERROR            PET0766 src/Superstructure/AttributeAPI/interface/ESMF_Attribute.F90:46022 ESMF_AttributeRemoveAttPackGrid( Not found  - Internal subroutine call returned Error
20230907 125011.127 ERROR            PET0766 ESMCI_Info.C:668 Info::erase() Not found  - [json.exception.out_of_range.403] key 'GridCornerLats:' not found
20230907 125011.127 ERROR            PET0766 ESMCI_Info.C:688 Info::erase() Not found  - Internal subroutine call returned Error
20230907 125011.127 ERROR            PET0766 ESMC_InfoCDef.C:243 ESMC_InfoErase() Not found  - Internal subroutine call returned Error
20230907 125011.127 ERROR            PET0766 ESMF_Info.F90:2656 ESMF_InfoRemove() Not found  - Internal subroutine call returned Error
20230907 125011.127 ERROR            PET0766 src/Superstructure/AttributeAPI/interface/ESMF_Attribute.F90:46022 ESMF_AttributeRemoveAttPackGrid( Not found  - Internal subroutine call returned Error
20230907 125021.303 INFO             PET0766 Model Advance: before wrtcomp run

To Reproduce:

compile and run the feature/add_c5 branch using DEBUG for the bmark P8 test.

Additional context

Output

[ulmononian]

i had 100% of tests pass about a week ago or so on c5. so the debug test runs to completion here but you are seeing these errors anyway?

[DeniseWorthen]

Yes, the test runs, so it is not fatal. I don't know what it means actually. It must be in MAPL?

[natalie-perlin]

@DeniseWorthen @ulmononian - does spack-stack need to have esmf-debug module for running the cpld_control_nowave_noaero_p8 and cpld_bmark_p8 in debug mode ?

[DeniseWorthen]

No, we no longer use esmf built w/ debug.

[zach1221]

Hi, @DeniseWorthen . Below are my experiment directories for cpld_control_nowave_noaero_p8_intel & cpld_bmark_p8_intel on Gaea C5, if you want to take a look at the PET logs. If you dont have access to view let me know.

/lustre/f2/scratch/Zachary.Shrader/FV3_RT/rt_76751/cpld_control_nowave_noaero_p8_intel /lustre/f2/scratch/Zachary.Shrader/FV3_RT/rt_76751/cpld_bmark_p8_intel

[DeniseWorthen]

@zach1221 The same behaviour appears to be present. No PET error messages in the cpld_control_nowave_noaero_p8_intel test, which does not include aerosols. In the cpld_bmark_p8_intel test, the only PET log which does not contain the error message is PET0767 (the last atm PET).

[natalie-perlin]

@DeniseWorthen @zach1221

• all RT tests passes for Gaea-c5 with spack-stack/1.5.0. Modulefiles and scripts for gaea-c5 were updated as continuation of the PR into the UFS weather model repo started by @ulmononian (ufs-community/ufs-weather-model#1784). A PR with the updates was created: https://github.com/ulmononian/ufs-weather-model/pull/13 into @ulmononian fork, the log RegressionTests_gaea-c5.log is attached to that PR.

[DeniseWorthen]

@zach1221 @jkbk2004 The behavior on hercules w/rt these two cases is the same on Hercules. That is, there is no ERROR on the test w/o aerosols, ie, cpld_control_noaero_p8_intel. There is an PET log error on all but the last ATM PE for the cpld_bmark_p8 test. I also checked the low-resolution case, cpld_control_p8_intel and it shows the same thing---ERROR on all but the last ATM PE. Note, Aerosols run on the same PEs as the ATM. I believe this most likely a MAPL issue and not ESMF.

[natalie-perlin]

@DeniseWorthen - A working directory on Gaea-c5 with the cpld_bmark_p8 test is /lustre/f2/scratch/ncep/Natalie.Perlin/FV3_RT/rt_221558/cpld_bmark_p8_intel In case you may take a look to see whether this is the same behavior.

[DeniseWorthen]

@natalie-perlin I had checked earlier w/ @zach1221 run directories and confirmed that the same error message is present as in the original issue. I also later confirmed that the same message is present on a hercules run so it isn't specific to C5. I believe it to be a MAPL issue most likely.

[jkbk2004]

I think GSFC people might be able to access to hercules. It will be a great starting point if we can set up a MAPL debugging installation and experiment. @mathomp4 Is Jiang still available?

[mathomp4]

I think GSFC people might be able to access to hercules. It will be a great starting point if we can set up a MAPL debugging installation and experiment. @mathomp4 Is Jiang still available?

@jkbk2004 Yes. @weiyuan-jiang should be able to help. That said, I'm not sure he (or any of us) have access to Hercules. We do have access to Orion.

[weiyuan-jiang]

I think I have access to Hercules. Judging from the logging error, it is more like a problem from ESMF ( due to the building ? ). But I will take a look at it.

[DeniseWorthen]

@weiyuan-jiang Thanks for checking. The reason I suspect MAPL is that we do not see the error in cases w/o the aerosol component.

[weiyuan-jiang]

What version of gocart and MAPL are you using? @DeniseWorthen

[DeniseWorthen]

I'm not sure how to tell which version of gocart we are using, but for MAPL we are using 2.35.2-esmf-8.4.2

[jkbk2004]

GOCART hash: https://github.com/GEOS-ESM/GOCART/tree/041422934cae1570f2f0e67239d5d89f11c6e1b7

[zach1221]

Looks like 2.1.1 for gocart

[weiyuan-jiang]

I have talked to Ben. He thought there might be a small chance that an attribute is not there in the MAPL_GetHorzIJIndex call. ( It is confusing though, because it should not produce error) . Anyway, I have replaced that call in the branch . Would you please try this new MAPL?

[DeniseWorthen]

Since this error is not specific to C5 (it also shows up on Hercules), I've edited the issue title.