netCDF-4 files in inputdata/

samsrabin commented 1 year ago

Brief summary of bug

CESM seems unable to read input files in the netCDF-4 format. I'm not sure if that's something we want to fix, and if so, it'd probably be more of an ESMF issue. On our side, though, I noticed that some of our inputdata/ files (as well as some in shared/) are actually netCDF-4.

General bug information

CTSM version you are using: ctsm5.1.dev131

Does this bug cause significantly incorrect results in the model's science? Causes crashes.

Configurations affected: Issue encountered with IHistClm51BgcCrop. Other configurations also likely affected.

Details of bug

(Note that some of these are probably not used in model runs, in which case netCDF-4 is probably fine. Others may be read by some part of the code that doesn't crash on netCDF-4 files for whatever reason.)

In /glade/p/cesmdata/cseg/inputdata/lnd/clm2/, the following directories each have at least one netCDF-4 file (I've excluded rawdata/):

isotopes/ (although these are "netCDF-4 classic model"; not sure if that works)
mappingdata/grids/
mappingdata/maps/
paramdata/
PTCLMmydatafiles*/
regional_datasets/
surfdata_map/NEON/
urbandata/ (also all "netCDF-4 classic model")

The format of a file can be checked with ncdump -hs $filename | grep "_Format". I checked all netCDFs like so:

for f in $(find . -name "*nc" | grep -v "rawdata" | sort); do
    result=$(ncdump -hs $f 2>/dev/null | grep "_Format" | grep "netCDF-4")
    [[ "${result}" != "" ]] && echo $f $result
done

There are also the following input files in /glade/p/cesmdata/cseg/inputdata/share/:

meshes/360x720_120830_ESMFmesh_c20210507b.nc
meshes/360x720_120830_ESMFmesh_c20210507.nc
meshes/C12_181018_ESMFmesh.nc
meshes/C192_181018_ESMFmesh.nc
meshes/C24_181018_ESMFmesh.nc
meshes/C384_181018_ESMFmesh.nc
meshes/C48_181018_ESMFmesh.nc
meshes/C5_181018_ESMFmesh.nc
meshes/ne120np4.pg3_esmf_mesh_c200415.nc
meshes/ne30pg2_ESMFmesh_c20200727.nc
meshes/ne30pg3_ESMFmesh_c20200821.nc
meshes/ne3np4_ESMFmesh_c230714.nc
meshes/T85_ESMFmesh.c220321.nc
meshes/tx0.1v2_211102_ESMFmesh.nc
meshes/tx0.1v3_210910_ESMFmesh.nc
meshes/tx0.1v3_211102_ESMFmesh.nc
meshes/tx1v1_211102_ESMFmesh.nc
scripgrids/mpaso.EC120to60.scrip.161109.nc
scripgrids/ne480np4_scrip_c200409.nc
scripgrids/tx0.1v2_211102.nc
scripgrids/tx0.1v3_211102.nc
scripgrids/tx0.66v1_SCRIP_180604.nc
scripgrids/tx0.66v1_SCRIP_190314.nc
scripgrids/tx1v1_211102.nc

Important output or errors that show the problem

From a half-degree run that crashes due to a netCDF-4 mesh file, log file PET11.ESMF_LogFile:

20230807 100248.399 ERROR            PET11 /glade/p/cesmdata/cseg/PROGS/build/58448/esmf-8.4.1b02/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:429 ESMCI_mesh_create_from_ESMFMesh_ The NetCDF Library returned an error  -  Unable to open existing file: /glade/p/cesmdata/cseg/inputdata/share/meshes/360x720_120830_ESMFmesh_c20210507.nc, (PIO/PNetCDF error = NetCDF: Attempt to use feature that was not turned on when netCDF was built.)
20230807 100248.437 ERROR            PET11 /glade/p/cesmdata/cseg/PROGS/build/58448/esmf-8.4.1b02/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:670 ESMCI_mesh_create_from_ESMFMesh_ The NetCDF Library returned an error  - Internal subroutine call returned Error
20230807 100248.437 ERROR            PET11 /glade/p/cesmdata/cseg/PROGS/build/58448/esmf-8.4.1b02/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:284 ESMCI_mesh_create_from_file() The NetCDF Library returned an error  - Internal subroutine call returned Error
20230807 100248.437 ERROR            PET11 ESMCI_MeshCap.C:2594 MeshCap::meshcreatefromfilenew() The NetCDF Library returned an error  - Internal subroutine call returned Error
20230807 100248.437 ERROR            PET11 ESMF_Mesh.F90:1969 ESMF_MeshCreateFromFile() The NetCDF Library returned an error  - Internal subroutine call returned Error
20230807 100248.437 ERROR            PET11 dshr_mod.F90:347 The NetCDF Library returned an error  - Passing error in return code
20230807 100248.437 ERROR            PET11 atm_comp_nuopc.F90:422 The NetCDF Library returned an error  - Passing error in return code
20230807 100248.437 ERROR            PET11 /glade/p/cesmdata/cseg/PROGS/build/58448/esmf-8.4.1b02/src/Infrastructure/Trace/src/ESMCI_Trace.C:1527 ESMCI:TraceEventPhaseExit() Wrong argument specified  - Trace regions not properly nested exiting from region: [ATM] IPDv01p3 Expected exit from: datm_strdata_init
20230807 100248.437 ERROR            PET11 /glade/p/cesmdata/cseg/PROGS/build/58448/esmf-8.4.1b02/src/Infrastructure/Trace/src/ESMCI_Trace.C:1485 ESMCI::TraceEventCompPhaseExit() Wrong argument specified  - Internal subroutine call returned Error
20230807 100248.437 ERROR            PET11 ESMCI_FTable.C:832 ESMCI_FTableCallEntryPointVMHop Wrong argument specified  - Internal subroutine call returned Error
20230807 100248.438 ERROR            PET11 ESMCI_FTable.C:1100 c_esmc_compwait Wrong argument specified  - Internal subroutine call returned Error
20230807 100248.438 ERROR            PET11 ESMF_Comp.F90:1256 ESMF_CompExecute Wrong argument specified  - Internal subroutine call returned Error
20230807 100248.438 ERROR            PET11 ESMF_GridComp.F90:1426 ESMF_GridCompInitialize Wrong argument specified  - Internal subroutine call returned Error
20230807 100248.438 ERROR            PET11 ESM0001:src/addon/NUOPC/src/NUOPC_Driver.F90:2832 Wrong argument specified  - Failed calling phase 'IPDv01p3' Initialize for modelComp 2: ATM
20230807 100248.438 ERROR            PET11 ESM0001:src/addon/NUOPC/src/NUOPC_Driver.F90:1950 Wrong argument specified  - Passing error in return code
20230807 100248.438 ERROR            PET11 /glade/p/cesmdata/cseg/PROGS/build/58448/esmf-8.4.1b02/src/Infrastructure/Trace/src/ESMCI_Trace.C:1527 ESMCI:TraceEventPhaseExit() Wrong argument specified  - Trace regions not properly nested exiting from region: [ESM0001] IPDv02p3 Expected exit from: datm_strdata_init
20230807 100248.438 ERROR            PET11 /glade/p/cesmdata/cseg/PROGS/build/58448/esmf-8.4.1b02/src/Infrastructure/Trace/src/ESMCI_Trace.C:1485 ESMCI::TraceEventCompPhaseExit() Wrong argument specified  - Internal subroutine call returned Error
20230807 100248.438 ERROR            PET11 ESMCI_FTable.C:832 ESMCI_FTableCallEntryPointVMHop Wrong argument specified  - Internal subroutine call returned Error
20230807 100248.438 ERROR            PET11 ESMCI_FTable.C:1100 c_esmc_compwait Wrong argument specified  - Internal subroutine call returned Error
20230807 100248.438 ERROR            PET11 ESMF_Comp.F90:1256 ESMF_CompExecute Wrong argument specified  - Internal subroutine call returned Error
20230807 100248.438 ERROR            PET11 ESMF_GridComp.F90:1426 ESMF_GridCompInitialize Wrong argument specified  - Internal subroutine call returned Error
20230807 100248.438 ERROR            PET11 ensemble:src/addon/NUOPC/src/NUOPC_Driver.F90:2832 Wrong argument specified  - Failed calling phase 'IPDv02p3' Initialize for modelComp 1: ESM0001
20230807 100248.438 ERROR            PET11 ensemble:src/addon/NUOPC/src/NUOPC_Driver.F90:1955 Wrong argument specified  - Passing error in return code
20230807 100248.438 ERROR            PET11 ensemble:src/addon/NUOPC/src/NUOPC_Driver.F90:481 Wrong argument specified  - Passing error in return code
20230807 100248.438 ERROR            PET11 /glade/p/cesmdata/cseg/PROGS/build/58448/esmf-8.4.1b02/src/Infrastructure/Trace/src/ESMCI_Trace.C:1527 ESMCI:TraceEventPhaseExit() Wrong argument specified  - Trace regions not properly nested exiting from region: [ensemble] Init 1 Expected exit from: datm_strdata_init
20230807 100248.438 ERROR            PET11 /glade/p/cesmdata/cseg/PROGS/build/58448/esmf-8.4.1b02/src/Infrastructure/Trace/src/ESMCI_Trace.C:1485 ESMCI::TraceEventCompPhaseExit() Wrong argument specified  - Internal subroutine call returned Error
20230807 100248.438 ERROR            PET11 ESMCI_FTable.C:832 ESMCI_FTableCallEntryPointVMHop Wrong argument specified  - Internal subroutine call returned Error
20230807 100248.438 ERROR            PET11 ESMCI_FTable.C:1100 c_esmc_compwait Wrong argument specified  - Internal subroutine call returned Error
20230807 100248.438 ERROR            PET11 ESMF_Comp.F90:1256 ESMF_CompExecute Wrong argument specified  - Internal subroutine call returned Error
20230807 100248.438 ERROR            PET11 ESMF_GridComp.F90:1426 ESMF_GridCompInitialize Wrong argument specified  - Internal subroutine call returned Error
20230807 100248.438 ERROR            PET11 esmApp.F90:130 Wrong argument specified  - Passing error in return code
20230807 100248.438 INFO             PET11 Finalizing ESMF
20230807 100248.438 ERROR            PET11 /glade/p/cesmdata/cseg/PROGS/build/58448/esmf-8.4.1b02/src/Infrastructure/Trace/src/ESMCI_Trace.C:1636 ESMCI::TraceEventRegionExit() Wrong argument specified  - Trace regions not properly nested exiting from region: [ESMF] Expected exit from: datm_strdata_init
20230807 100248.438 ERROR            PET11 /glade/p/cesmdata/cseg/PROGS/build/58448/esmf-8.4.1b02/src/Infrastructure/Trace/src/ESMCI_Trace.C:1077 ESMCI::TraceClose() Wrong argument specified  - Internal subroutine call returned Error

ekluzek commented 1 year ago

Thanks for exploring this @samsrabin, this is very helpful. The rimport tool that we use to import files into svn inputdata doesn't allow files to be entered that are in NetCDF4 format. That lends me to believe these files must either NOT be in svn inputdata, OR they were entered in before we realized this restriction. So it would be good to figure out which is the case for the specific files.

We could ask in CSEG if supporting NetCDF-4 is even something we care about. But, I've always just converted them to CDF5 using nccopy when I find them.

billsacks commented 1 year ago

From discussion this morning:

We're curious if these are in the inputdata repository (and if they're actually referenced anywhere, e.g., in ccs_config)
Can also use ncdump -k
Could delete these NetCDF-4 files on glade

samsrabin commented 1 year ago

Of the 91 netCDF-4 files in lnd/clm2/mappingdata/, 48 are mentioned in at least one of bld/namelist_files/namelist_defaults_ctsm_tools.xml, bld/namelist_files/namelist_defaults_ctsm.xml, and tools/mksurfdata_map/mksurfdata_map.namelist. The mentioned files are listed in /glade/p/cesmdata/cseg/inputdata/lnd/clm2/netCDF4_files_refd. No other netCDF-4 files in lnd/clm2/ seem to be mentioned anywhere in an up-to-date CTSM repo with all externals checked out.

Of the 24 netCDF-4 files in share/, 5 are mentioned in ccs_config/component_grids_nuopc.xml:

meshes/360x720_120830_ESMFmesh_c20210507.nc
meshes/C192_181018_ESMFmesh.nc
meshes/C24_181018_ESMFmesh.nc
meshes/C384_181018_ESMFmesh.nc
meshes/C48_181018_ESMFmesh.nc

samsrabin commented 1 year ago

None of the netCDF-4 files appear to be in the SVN repository.

ESCOMP / CTSM