V3 fix folder for test/develop

chengdang commented 1 year ago

With the progressive merge of v2.4.1 into v3, and continuous generation of coefficients for new sensors, this issue tracks the update history of fix folder for testing/developing

chengdang commented 1 year ago

fix_REL-2.4.1_20221109

v2.4.1 aerosol coefficients (the most recent)
v2.4.1 emissivity coefficients (the most recent, not compatible with the v3 develop yet)
coefficients for new sensors (the most recent)

chengdang commented 1 year ago

fix_REL-3_20230303

temporarily placed in google drive: fix_REL-3_20230303.tar.gz with restricted access

v2.4.1 aerosol coefficients (the most recent version)
v2.4 emissivity coefficients (updates on surface emissivity codes are not merged yet)
coefficients for new sensors (the most recent version)

BenjaminTJohnson commented 1 year ago

@chengdang I'm currently also working on assembling all of the transmittance coefficients as well. When you have this ready, I will merge it with what I'm currently doing, and will run some tests to ensure we're using the latest versions across all coefficients.

BenjaminTJohnson commented 1 year ago

@chengdang one other thing: netCDF directories should be "netCDF", rather than "netcdf".

chengdang commented 1 year ago

@BenjaminTJohnson sounds good! I’ll update this link once the upload is done. (it takes my wifi 3 hours to upload 4GB now )

chengdang commented 1 year ago

@chengdang one other thing: netCDF directories should be "netCDF", rather than "netcdf".

Oh yes, is this for netcdf directories in the fix folder? I think all of them are netCDF now.

chengdang commented 1 year ago

Google drive link updated for fix_REL-3_20230303.tar.gz with restricted access.

BenjaminTJohnson commented 1 year ago

@chengdang

In comparing the emissivity between what I thought was correct and what you've got in your tarball, Nalli, Nalli2, and WuSmith are different:

Only in ../fix_crtmv3-develop/fix/EmisCoeff/IR_Water/Big_Endian: Nalli2.IRwater.EmisCoeff.bin
Files fix_REL-3_20230303/fix/EmisCoeff/IR_Water/Big_Endian/Nalli.IRwater.EmisCoeff.bin and ../fix_crtmv3-develop/fix/EmisCoeff/IR_Water/Big_Endian/Nalli.IRwater.EmisCoeff.bin differ
Files fix_REL-3_20230303/fix/EmisCoeff/IR_Water/Big_Endian/WuSmith.IRwater.EmisCoeff.bin and ../fix_crtmv3-develop/fix/EmisCoeff/IR_Water/Big_Endian/WuSmith.IRwater.EmisCoeff.bin differ
Only in ../fix_crtmv3-develop/fix/EmisCoeff/IR_Water/Little_Endian: Nalli2.IRwater.EmisCoeff.bin
Files fix_REL-3_20230303/fix/EmisCoeff/IR_Water/Little_Endian/Nalli.IRwater.EmisCoeff.bin and ../fix_crtmv3-develop/fix/EmisCoeff/IR_Water/Little_Endian/Nalli.IRwater.EmisCoeff.bin differ
Files fix_REL-3_20230303/fix/EmisCoeff/IR_Water/Little_Endian/WuSmith.IRwater.EmisCoeff.bin and ../fix_crtmv3-develop/fix/EmisCoeff/IR_Water/Little_Endian/WuSmith.IRwater.EmisCoeff.bin differ

For example, looks like there are two versions of the Nalli.IRwater.EmisCoeff (this one):

2070cb8728d5dd4f302af7cfb9b0527bd9d9e464  ./crtm-internal_REL-2.4.0_emc/fix/EmisCoeff/IR_Water/Little_Endian/Nalli.IRwater.EmisCoeff.bin
2070cb8728d5dd4f302af7cfb9b0527bd9d9e464  ./fix_crtm-internal_develop/EmisCoeff/IR_Water/Little_Endian/Nalli.IRwater.EmisCoeff.bin
2070cb8728d5dd4f302af7cfb9b0527bd9d9e464  ./fix_REL-3_20230303/fix/EmisCoeff/IR_Water/Little_Endian/Nalli.IRwater.EmisCoeff.bin

and this one:

65a1c4b50495211062d27ebdf60b72de4fd35fcf  ./fix_REL-2.4.1_20221109/fix/EmisCoeff/IR_Water/Little_Endian/Nalli.IRwater.EmisCoeff.bin

Same for Nalli2 and WuSmith.

Plese verify which shasum (on the little endian file) or file sizes are the correct ones for Nalli, Nalli2, and WuSmith.

Thanks.

chengdang commented 1 year ago

Hmm, I see. Thank you for the comparison!

Nalli2.IRwater.EmisCoeff.bin is not included in fix_REL-3_20230303 because it is not supported in the current v3 develop branch.
It's possible that there are more than one version of these coefficient files during development, especially with the update of emissivity scheme and version/release numbers.
The latest emissivity coefficient files I updated were uploaded as folder fix_REL-2.4.1_20220708.tar.gz, and I assumed fix_REL-2.4.1_20221109.tar.gz was built on top it with the latest emissivity files, but I did not check.
Was fix_crtmv3-develop the original fix folder from Mark's branch?

I'll look more into this next week too!

BenjaminTJohnson commented 1 year ago

I think that answers my question. I'll use the ones consistent with REL-2.4.1. Thanks.

BenjaminTJohnson commented 1 year ago

The full v3.0.0 tarball is at (BE/LE/netCDF):

ftp.ssec.wisc.edu/pub/s4/CRTM/fix_REL-3.0.0_20230303.tgz

Information regarding the v3.0.0_skylab_4.0.tar.gz file for JEDI/UFO (LE/netCDF only)

ftp.ssec.wisc.edu/pub/s4/CRTM/crtm_coefficients_3.0.0_skylab_4.0.tar.gz

It contains the following files and instruments (subsets and band-specific variants removed from this list, but are included in the tarball):

For now, we're still including little endian binary format, and adding/converting netCDF as we go. The transition to "all netCDF" will have to be gradual, likely in chunks of effort.

structure: crtm/3.0.0_skylab_4.0/

new/updated surface files: Nalli2.IRwater.EmisCoeff.bin (supports future update, don't use yet) Nalli.IRwater.EmisCoeff.bin (replaces existing) WuSmith.IRwater.EmisCoeff.bin (replace existing) EmisCoeff/IR_Snow/Nalli/* (supports future update, don't use yet)

Coefficients: AerosolCoeff/Little_Endian/AerosolCoeff.bin (updated)

Experimental Cloud coefficient files for microphysical consistency testing: CloudCoeff.GFDLFV3.-109z-1.bin cloudcoeff_GFDLFV3.bin (capitalization on these will be fixed in the future) CloudCoeff_MW_LE_2.4.0_a.bin CloudCoeff_MW_LE_2.4.0_exp.bin CloudCoeff.Thompson08.-109z-1.bin cloudcoeff_Thompson08.bin CloudCoeff.WSM6.-109z-1.bin cloudcoeff_WSM6.bin CloudCoeff.WSM6.MieSphere.bin

satellite sensors:

aatsr_envisat abi-81K_g17 abi_g16 abi_g17 abi_g18 abi_gr ahi_himawari8 ahi_himawari9 airs_aqua airs_g13 ami_gk2 amsr2_gcom-w1 amsre_aqua amsua_aqua amsua_metop-a amsua_metop-a_v2 amsua_metop-b amsua_metop-c amsua_metop-c_v2 amsua_n15 amsua_n16 amsua_n17 amsua_n18 amsua_n19 amsub_n15 amsub_n16 amsub_n17 aster_terra atms_j2 atms_j2-srf atms_n20 atms_n20-srf atms_n21 atms_n21-srf atms-ng_v1 atms_npp atsr1_ers1 atsr2_ers2 avhrr2_n06 avhrr2_n07 avhrr2_n08 avhrr2_n09 avhrr2_n10 avhrr2_n11 avhrr2_n12 avhrr2_n14 avhrr2_tirosn avhrr3_metop-a avhrr3_metop-b avhrr3_metop-c avhrr3_n15 avhrr3_n16 avhrr3_n17 avhrr3_n18 avhrr3_n19 cowvr_ors6cpr_cloudsat cris-fsr_j2 cris-fsr_n20 cris-fsr_n21 cris-fsr_npp cris_n20 (I don't immediately recall what the difference is vs -fsr) cris_nppdpr_gpm eon_mw.v1 geostorm_proposed.boxcar geostorm_proposed.gaussian giirsB1_fsr_fy4a giirsB2_fsr_fy4a gmi_gpm hamsr_grip hirs2_n06 hirs2_n07 hirs2_n08 hirs2_n09 hirs2_n10 hirs2_n11 hirs2_n12 hirs2_n14 hirs2_tirosn hirs3_n15 hirs3_n16 hirs3_n17 hirs4_metop-a hirs4_metop-b hirs4_n18 hirs4_n19 hsb_aqua iasi_g13 iasi_metop-a iasi_metop-b iasi_metop-c imgr_g08 imgr_g09 imgr_g10 imgr_g11 imgr_g12 imgr_g13 imgr_g14 imgr_g15 imgr_insat3d imgr_mt1r imgr_mt2 iras_fy3a iras_fy3b ivissr_fy2c ivissr_fy2d ivissr_fy2e ivissr_fy2f madras_meghat masc_cubesat mersi_fy3a mhs_metop-a mhs_metop-b mhs_metop-c mhs_metop-c_v2 mhs_n18 mhs_n19 micromas_cs00 micromas_cs01 micromas_cs02 micromas_cs03 micromas_cs04 micromas_cs05 mi-l_coms mi-l_coms.v2 mi-m_coms miras_smos modis_aqua modis_terra msu_n06 msu_n07 msu_n08 msu_n09 msu_n10 msu_n11 msu_n12 msu_n14 msu_tirosn mwhs_fy3a mwhs_fy3b mwi_eps-sg_v1 mwri_fy3a mwri_fy3b mwts_fy3a mwts_fy3b radiometer_smap saphir_meghat seviri_m08 seviri_m09 seviri_m10 seviri_m11 slstr_sentinel3a sndr_g08 sndr_g09 sndr_g10 sndr_g11 sndr_g12 sndr_g13 sndr_g14 sndr_g15 sndr_insat3d ssmi_f08 ssmi_f10 ssmi_f11 ssmi_f13 ssmi_f14 ssmi_f15 ssmis_f16 ssmis_f17 ssmis_f18 ssmis_f19 ssmis_f20 ssmt1_f13 ssmt1_f15 ssmt2_f14 ssmt2_f15 ssu_n06 ssu_n07 ssu_n08 ssu_n09 ssu_n11 ssu_n14 ssu_pseudo ssu_tirosn tempest-D_cubesat tmi_trmm tropics_designed_v1 tropics_sv1_srf_v3 u.omps-npAllFOV_j2 u.omps-tcAllFOV_j2 v.abi_g16 v.abi_g17 v.abi_g18 (untested) v.abi_gr v.ahi_himawari8 v.ahi_himawari9 vas_g04 vas_g05 vas_g06 vas_g07 v.aster_terra v.avhrr2_n14 v.avhrr3_metop-a v.avhrr3_metop-b v.avhrr3_n15 v.avhrr3_n16 v.avhrr3_n17 v.avhrr3_n18 v.avhrr3_n19 vhrr_kalpana1 viirs-i_j1 viirs-i_j2 viirs-i_n21 viirs-i_npp viirs-m_j1 viirs-m_j2 viirs-m_n21 viirs-m_npp v.imgr_g11 v.imgr_g12 v.imgr_g13 v.imgr_g14 v.imgr_g15 v.imgr_mt2 v.iras_fy3a v.iras_fy3b virr_fy3a vissrDetA_gms5 vissrDetB_gms5 v.ivissr_fy2c v.ivissr_fy2d v.ivissr_fy2e v.ivissr_fy2f v.mi-l_coms v.mi-m_coms v.modis_aqua v.modis_terra v.seviri_m08 v.seviri_m09 v.seviri_m10 v.sndr_g08 v.sndr_g09 v.sndr_g10 v.sndr_g11 v.sndr_g12 v.sndr_g13 v.sndr_g14 v.sndr_g15 vtprS1_itos vtprS2_itos vtprS3_itos vtprS4_itos v.viirs-dnb_j1 v.viirs-i_j1 v.viirs-i_j2 (we need to add the n21 version of these, it will be exactly the same though) v.viirs-i_npp v.viirs-m_j1 v.viirs-m_j2 v.viirs-m_npp windsat_coriolis

Also, we'll have a look and see why these are different, for now let us know if you spot anything unusual with amsua_metop-b in particular. Files 3.0.0_skylab_4.0/SpcCoeff/Little_Endian/amsua_metop-b.SpcCoeff.bin and 2.4.0_skylab_3.0/SpcCoeff/Little_Endian/amsua_metop-b.SpcCoeff.bin differ Files 3.0.0_skylab_4.0/SpcCoeff/Little_Endian/hirs4_metop-a.SpcCoeff.bin and 2.4.0_skylab_3.0/SpcCoeff/Little_Endian/hirs4_metop-a.SpcCoeff.bin differ Files 3.0.0_skylab_4.0/SpcCoeff/Little_Endian/hirs4_n18.SpcCoeff.bin and 2.4.0_skylab_3.0/SpcCoeff/Little_Endian/hirs4_n18.SpcCoeff.bin differ Files 3.0.0_skylab_4.0/SpcCoeff/Little_Endian/hirs4_n19.SpcCoeff.bin and 2.4.0_skylab_3.0/SpcCoeff/Little_Endian/hirs4_n19.SpcCoeff.bin differ

Tropics sv1_srf_v1 was removed, replaced by the v3 version. Only in 2.4.0_skylab_3.0/SpcCoeff/Little_Endian: tropics_sv1_srf_v1.SpcCoeff.bin Only in 2.4.0_skylab_3.0/TauCoeff/ODPS/Little_Endian: tropics_sv1_srf_v1.TauCoeff.bin

BenjaminTJohnson commented 1 year ago

Here's a list of sensors with SpcCoeff files that are not able to be read in properly:

Big-Endian:

amsua_metop-b
atms_j2.BE
hamsr_grip
hirs2_n09
hirs2_n10
hirs2_n11
hirs2_n12
hirs2_n14
hirs2-UWS_n09
hirs2-UWS_n10
hirs2-UWS_n11
hirs2-UWS_n12
hirs2-UWS_n14
hirs3_n15
hirs3_n16
hirs3_n17
hirs3-UWS_n15
hirs3-UWS_n16
hirs3-UWS_n17
**hirs4_metop-a**
hirs4_n18
**hirs4_n19**
hirs4-UWS_metop-a
hirs4-UWS_n18
hirs4-UWS_n19
ssmis_f16.v3-eff_passband

and Little_Endian:

**amsua_metop-a_v2**
**amsua_metop-c_v2**
**cpr_cloudsat**
**dpr_gpm**
hamsr_grip
hirs2_n09
hirs2_n10
hirs2_n11
hirs2_n12
hirs2_n14
hirs2-UWS_n09
hirs2-UWS_n10
hirs2-UWS_n11
hirs2-UWS_n12
hirs2-UWS_n14
hirs3_n15
hirs3_n16
hirs3_n17
hirs3-UWS_n15
hirs3-UWS_n16
hirs3-UWS_n17
hirs4_n18
hirs4-UWS_metop-a
hirs4-UWS_n18
hirs4-UWS_n19
imgrD1S1_g13
imgrD1S2_g13
imgrD2S1_g13
imgrD2S2_g13
imgrS1_g13
imgrS2_g13
ssmis_f16.v3-eff_passband
**u.omps-npAllFOV_j2**
**u.omps-tcAllFOV_j2**
**v.avhrr2_n14**
**v.ivissr_fy2c**
**v.ivissr_fy2d**
**v.ivissr_fy2e**
**v.ivissr_fy2f**

** indicates differences between the two lists. the hirs4_n19 might be different because I copied a different LE file during testing -- I think naturally both are failing.

The common error message is:

SpcCoeff_ReadFile(Binary)(FAILURE) : Error reading channel data. I/O past end of record on unformatted file
 CRTM_SpcCoeff_Load(FAILURE) : Error reading SpcCoeff file #1, hirs2_n14.SpcCoeff.bin

Some are missing their respective TauCoeff files, so I will remove those.

I will continue working on ensuring that the appropriate files are in the tarball, and will provide an update here and via email once all tarball items are properly loading with v3, we'll have to decide later if they're producing reasonable results. This probably will not fix any adjoint problems.

StegmannJCSDA commented 1 year ago

u.omps from NESDIS is not expected to work.

JCSDA / CRTMv3

V3 fix folder for test/develop #18