CloudCoeff_Convert tool (bin to NC and NC to bin) creates incorrect netCDF file.

[BenjaminTJohnson]

binary to netcdf using CloudCoeff_Convert.f90 (src/Coefficients/CloudCoeff/CloudCoeff_Convert/), it produces a netcdf file that is mostly complete, but creates an extra N_IR_DENSITIES value. When the binary file specifies, for example N_IR_DENSITIES=4, the produced netcdf file has N_IR_DENSITIES=5, this creates some empty slots at the end of each structure that depends on N_IR_DENSITIES.

It's not immediately clear (from inspecting the relevant codes) where the "+1" is coming from.

[BenjaminTJohnson]

Proposed solution: similar to other coefficient utilities, use NC2BIN and BIN2NC versions of the utility. Currently only NC2BIN exists, and properly converts older CloudCoeff*nc4 files to an equivalent binary file.

However, CloudCoeff_NC2BIN produces an error at the end: "CloudCoeff_netCDF_to_Binary(SUCCESS) : CloudCoeff object comparison failed." The binary file appears to be perfectly intact, so I'll dig into the utility to sort out why it's claiming success but saying the object comparison failed...

[BenjaminTJohnson]

CloudCoeff_NC2BIN (branch: feature/btj_spccoeff_taucoeff_utilities) fails to create binary on CloudCoeff_DDA_Moradi_2022:

    Enter the INPUT netCDF CloudCoeff filename : CloudCoeff_DDA_Moradi_2022.nc4

     Enter the OUTPUT Binary CloudCoeff filename: CloudCoeff_DDA_Moradi_2022.bin 
 CloudCoeff_ReadFile(netCDF)(INFORMATION) : FILE: CloudCoeff_DDA_Moradi_2022.nc4; 
 CloudCoeff RELEASE.VERSION:   3.004  N_FREQUENCIES(MW)= 200  N_FREQUENCIES(IR)=  61  N_RADII(MW)=**  N_RADII(IR)=10  N_TEMPERATURES= 8  N_MW_DENSITIES=18  N_IR_DENSITIES= 4  N_LEGENDRE_TERMS=38  N_PHASE_ELEMENTS= 1
 CloudCoeff_WriteFile(Binary)(INFORMATION) : FILE: CloudCoeff_DDA_Moradi_2022.bin; 
 CloudCoeff RELEASE.VERSION:   3.004  N_FREQUENCIES(MW)= 200  N_FREQUENCIES(IR)=  61  N_RADII(MW)=**  N_RADII(IR)=10  N_TEMPERATURES= 8  N_MW_DENSITIES=18  N_IR_DENSITIES= 4  N_LEGENDRE_TERMS=38  N_PHASE_ELEMENTS= 1
 CloudCoeff_ReadFile(Binary)(FAILURE) : Error reading infrared data. IOSTAT = 67
 CloudCoeff_netCDF_to_Binary(FAILURE) : Error reading Binary file CloudCoeff_DDA_Moradi_2022.bin for test
 CloudCoeff_NC2BIN(FAILURE) : CloudCoeff netCDF -> Binary conversion failed!

[BenjaminTJohnson]

    Enter the INPUT netCDF CloudCoeff filename : CloudCoeff.nc4

     Enter the OUTPUT Binary CloudCoeff filename: CloudCoeff.bin
 CloudCoeff_ReadFile(netCDF)(INFORMATION) : FILE: CloudCoeff.nc4; 
 CloudCoeff RELEASE.VERSION:   3.004  N_FREQUENCIES(MW)=  31  N_FREQUENCIES(IR)=  61  N_RADII(MW)=10  N_RADII(IR)=10  N_TEMPERATURES= 5  N_MW_DENSITIES= 3  N_IR_DENSITIES= 4  N_LEGENDRE_TERMS=38  N_PHASE_ELEMENTS= 1
 CloudCoeff_WriteFile(Binary)(INFORMATION) : FILE: CloudCoeff.bin; 
 CloudCoeff RELEASE.VERSION:   3.004  N_FREQUENCIES(MW)=  31  N_FREQUENCIES(IR)=  61  N_RADII(MW)=10  N_RADII(IR)=10  N_TEMPERATURES= 5  N_MW_DENSITIES= 3  N_IR_DENSITIES= 4  N_LEGENDRE_TERMS=38  N_PHASE_ELEMENTS= 1
 CloudCoeff_ReadFile(Binary)(INFORMATION) : FILE: CloudCoeff.bin; 
 CloudCoeff RELEASE.VERSION:   3.004  N_FREQUENCIES(MW)=  31  N_FREQUENCIES(IR)=  61  N_RADII(MW)=10  N_RADII(IR)=10  N_TEMPERATURES= 5  N_MW_DENSITIES= 3  N_IR_DENSITIES= 4  N_LEGENDRE_TERMS=38  N_PHASE_ELEMENTS= 1
 CloudCoeff_netCDF_to_Binary(SUCCESS) : CloudCoeff object comparison failed.

However, produces results that are identical to the existing LE CloudCoeff.bin file.

[BenjaminTJohnson]

The newly created CloudCoeff_BIN2NC has the same problem as CloudCoeff_Convert. Suggests that there's a problem in the binary to netcdf library.

[BenjaminTJohnson]

Converted CloudCoeff_DDA_Moradi_2022.nc4 using CloudCoeff_NC2BIN:

[s4-submit] /data/users/bjohnson/CRTM/CRTMv3_REL-3.1.1/build/test_data/3.1.1/fix_REL-3.1.1.0/fix/CloudCoeff/netCDF> ./CloudCoeff_Inspect 

     **********************************************************
                         CloudCoeff_Inspect

      Program to display the contents of a CRTM Binary format 
      CloudCoeff file to stdout.

      $Revision$
     **********************************************************

     Enter the Binary CloudCoeff filename: CloudCoeff_DDA_Moradi_2022.bin
 CloudCoeff_ReadFile(Binary)(FAILURE) : Error reading infrared data. IOSTAT = 67
 CloudCoeff_Inspect(FAILURE) : Error reading Binary CloudCoeff file CloudCoeff_DDA_Moradi_2022.bin

Most likely related to the n_IR_Densities+1 logic between CloudCoeff_IO.f90 and CloudCoeff_netCDF_IO.f90. One issue that I see is that in some places n_IR_Densitiies is set equal to n_MW_Densities, but clearly this isn't true for the present CloudCoeff file.

A workable solution, starting with CRTM v3.2 would be to default to netCDF for the cloud coefficient files.

[BenjaminTJohnson]

@imoradi there is an issue in the CloudCoeff_IO and/or CloudCoeff_netCDF_IO that are manifesting the CloudCoeff_BIN2NC routines.

To test, run CloudCoeff_BIN2NC on an existing CloudCoeff.bin (the default file), and then notice that the n_IR_densities is 5 in the produced file, it should be 4. There's a few +1's in the CloudCoeff_IO codes regarding this value, but I wasn't able to figure out the right combination to fix the issue.

[imoradi]

@BenjaminTJohnson all the cloud coefficients are available in NetCDF so why do we need to carry out this capability? All the future CloudCoeff files can be simply created in NetCDF. I understand the need for having the netcdf conversion tools for the Tau and Spc coefficients.

[BenjaminTJohnson]

@imoradi I was concerned about there being an actual bug in the IO routines that could affect future netCDF read/write.

[imoradi]

I have tried the NetCDF readers with different CloudCoeff file and all work fine. This has something to do with the fact that old cloudcoeff used to index I believe IR from 0 but MW from 1 but the netcdf readers work fine.

[BenjaminTJohnson]

@imoradi okay -- let's ignore it for now. I think it's also time to consider how we structure the coefficients when we're not restricted by the binary formats.