search

esmf-org / esmf

The Earth System Modeling Framework (ESMF) is a suite of software tools for developing high-performance, multi-component Earth science modeling applications.
https://earthsystemmodeling.org/
Other
150 stars 70 forks source link

Ensure that meta data keeps their storage typekind through Attribute/Info #129

Open theurich opened 1 year ago

theurich commented 1 year ago

This is a request to look through the existing ESMF_Attribute, and/or ESMF_Info unit testing to make sure we successfully test that the typekind of meta data is kept in tact from when it is set to where it is retrieved. If testing is incomplete, it needs to be extended.

This request is triggered by issue https://github.com/ufs-community/ufs-weather-model/issues/1621, specifically comment https://github.com/ufs-community/ufs-weather-model/issues/1621#issuecomment-1450631888 that seems to indicate that in the latest version of ESMF (v8.4.1b07, but likely also on develop) integer attributes set on a FieldBundle show up as strings in the NetCDF file:

1109,1113c1109,1113
<               :grid_id = 1s ;
<               :imp_physics = 11s ;
<               :landsfcmdl = 1s ;
<               :ncld = 11s ;
<               :nsoil = 4s ;
---
>               :grid_id = 1 ;
>               :imp_physics = 11 ;
>               :landsfcmdl = 1 ;
>               :ncld = 11 ;
>               :nsoil = 4 ;

The comparison here between the two outputs is v8.4.1b07 vs v8.3.0b09. A relevant ESMF commit between those two versions is likely https://github.com/esmf-org/esmf/commit/6906b7b22641cfd46c42c1388f30f31feaf45280. But there may be others.

There is a lot going on from where the Attributes are being set in the FV3 write component (https://github.com/NOAA-EMC/fv3atm/blob/develop/io/FV3GFS_io.F90#L3467) to where they are written to NetCDF in side the ESMF code (https://github.com/esmf-org/esmf/blob/c55cd8ec6b3e9392352dad59cd63ffc8268d024b/src/Infrastructure/IO/src/ESMCI_PIO_Handler.C#L1423). It involves the deprecated ESMF_Attribute layer, but that is still used here for the I/O Attributes, it involves the ESMF_Info implementation of the meta data handling, and then finally the I/O layer to write the attributes to NetCDF. Review and extension of testing on all those levels might be needed.

theurich commented 1 year ago

A few more pieces of information:

  1. The 's' suffix actually stands for "short integer" (see http://www.bic.mni.mcgill.ca/users/sean/Docs/netcdf/guide.txn_77.html)! So that is actually correct here, and indicates that v8.4.1 (and presumably the develop branch of ESMF) correctly preserves the typekind of the metadata, all the way into being written into NetCDF file.
  2. Jun reports that she only sees the 's' suffix on the cubed sphere files written through call to ESMFproto_FieldBundleWrite(), and not for files written directly via NetCDF calls (https://github.com/ufs-community/ufs-weather-model/issues/1621#issuecomment-1452189598). This does make sense because ESMFproto_FieldBundleWrite() does go through the ESMF_FieldBundleWrite() method, which now strictly preserves the typekind of attributes.
  3. So the final conclusion here right now is that everything is working correctly inside of ESMF. The remaining action is to replace the custom ESMFproto_FieldBundleWrite() routine, which is located in the FV3 write component, with a direct call into the official ESMF_FieldBundleWrite() method, now that the latter is multi-tile enabled, as of ESMF 8.4.0.