Open emilyhcliu opened 3 weeks ago
Aerosol Coefficients
when Cloud Coefficients
are loadedThe above code block indicates that if the cloud coefficients are loaded, the aerosol coefficients need to be loaded as well. So, I modified GSI to load the aerosol coefficients for all-sky assimilation of ATMS and AMSU-A, for which the cloud coefficients are required. This resolved the issue of n_phase_elenemts
difference between cloud and aerosol coefficients.
Action required: Only GSI changes are required -- need to load aerosol coefficients when cloud coefficients are loaded
GSI develop + CRTMv2.4.0.1 for ATMS N21
GSI develop + CRTMv3.1.1 for ATMS N21
GSI develop + CRTMv2.4.0.1 for ATMS N20
GSI develop + CRTMv3 for ATMS N20
GSI develop + CRTMv2.4.0.1 for ATMS NPP
GSI develop + CRTMv3 for ATMS NPP
@emilyhcliu can you provide me with the endianness and the shasum of the atms_n21 coefficient that you're using here for v2.4.0 and v3.1.1?
Try copying the v2.4.0 coefficient into the v3.1.1 directory, replacing the one there (assuming it has a different shasum).
Here's a list of the shasums of the various CRTM releases for atms_n21 (big endian only):
8f8c227bb97e9560359363da795e63d090acc7a6 ./fix_REL-2.4.0_emc_07112023/fix/SpcCoeff/Big_Endian/atms_n21.SpcCoeff.bin
8f8c227bb97e9560359363da795e63d090acc7a6 ./fix_REL-3.0.1_20230923/fix/SpcCoeff/Big_Endian/atms_n21.SpcCoeff.bin
8f8c227bb97e9560359363da795e63d090acc7a6 ./fix_REL-3.1.0.1_01252024/fix/SpcCoeff/Big_Endian/atms_n21.SpcCoeff.bin
8f8c227bb97e9560359363da795e63d090acc7a6 ./fix_REL-3.1.0.3/fix/SpcCoeff/Big_Endian/atms_n21.SpcCoeff.bin
8f8c227bb97e9560359363da795e63d090acc7a6 ./fix_REL-3.1.1.0/fix/SpcCoeff/Big_Endian/atms_n21.SpcCoeff.bin
8f8c227bb97e9560359363da795e63d090acc7a6 ./fix_REL-3.1.1.1/fix/SpcCoeff/Big_Endian/atms_n21.SpcCoeff.bin
851124f6ea7773ab9acf931d2f853cd5c5913ebf ./crtm-internal_REL-2.4.0_emc/fix/SpcCoeff/Big_Endian/atms_n21.SpcCoeff.bin
It's possible I have mistakenly copied the wrong one from to the v3.x tarballs, i.e., I suspect now that you might need the last one in the list above, when I've been copying the first one to the various tarballs.
@BenjaminTJohnson Thanks for looking into N21. I am testing CRTMv3.1.1 and CRTMv2.4.0.1 on ORION The coefficients used are Little_Endian Here is the list of shasum for N21 from CRTMv3.1.1
82beb7dc1ccc9b6cb0a5a7e32904514cd79c7c09 atms_n21.SpcCoeff.bin
b587a277f15013a152cbc0748af52c5367f33c0c atms_n21.TauCoeff.bin
Here is the list of shasum for N21 from CRTMv2.4.0.1 (used in operational GFS)
8f8c227bb97e9560359363da795e63d090acc7a6 atms_n21.SpcCoeff.bin
bcfc9d95e808f11746f81d5d9d325278a4e898b0 atms_n21.TauCoeff.bin
@BenjaminTJohnson One question. Is there a Jacobian-related change in CRTMv3.1.1?
Okay, thanks. Is there any reason you're compiling in Little Endian, or was that the default?
I think that the problem is that the LE coefficient isn't the corresponding one to the BE coefficient -- EMC was the only one using the BE coefficient that was updated awhile back, so it looks like the corresponding LE coefficient wasn't made.
Two options (1) compile and run with BE coefficients... you can try the export trick:
export F_UFMTENDIAN=big
No need to recompile, just re-run the code and it will try to read BE instead of LE, but of course you'll need the appropriate soft links to the BE coefficients.
(2) I can create a LE coefficient for you from the BE coefficient. I will do this anyway, but it might take some time.
Okay, thanks. Is there any reason you're compiling in Little Endian, or was that the default?
I think that the problem is that the LE coefficient isn't the corresponding one to the BE coefficient -- EMC was the only one using the BE coefficient that was updated awhile back, so it looks like the corresponding LE coefficient wasn't made.
Two options (1) compile and run with BE coefficients... you can try the export trick:
export F_UFMTENDIAN=big
No need to recompile, just re-run the code and it will try to read BE instead of LE, but of course you'll need the appropriate soft links to the BE coefficients.
(2) I can create a LE coefficient for you from the BE coefficient. I will do this anyway, but it might take some time
The orion takes little endian by default. OK. I will try the export trick you suggest and run with big_endians. Thanks!
@emilyhcliu Okay, good to know. I will make a point now to verify that all LE files are the same as the BE files, which I should've done before!
I have converted the BE file to the LE file:
Please test it first: https://bin.ssec.wisc.edu/pub/s4/CRTM/file/atms_n21-LE.SpcCoeff.bin
Rename to atms_n21.SpcCoeff.bin
or point softlink with the appropriate name.
If it works, this will go into future tarball releases (3.1.1.2+)
@BenjaminTJohnson The big endian
N21 coefficient file is Good.
CRTMv3.1.1 + Big Endian coefficients
CRTMv2.4.0.1
@BenjaminTJohnson I am using Big_Endian coefficients with export F_UFMTENDIAN=big
on ORION
The ATMS N21 looks good now with the Big_Endian coefficients (see my previous comment)
I ran CRTMv3.1.1 wth Big_Endian coefficients for all other radiance types and ran into a problem for amsua_metop-b. Here is the error message:
SpcCoeff_ReadFile(Binary)(FAILURE) : Error reading channel data. input statement requires too much data, unit 10, file
/work2/noaa/da/eliu/ORION/GSIobserver_v16/allobs/2024021900/gsi/./crtm_coeffs/amsua_metop-b.SpcCoeff.bin
Here are the shasum info for amsua_metop-b (Big_Endian)
5cff45cd262710090729aec6fa69eb393969483e amsua_metop-b.SpcCoeff.bin
52217cd06c5bbc8f5d9837cdb027f810d4d60164 amsua_metop-b.TauCoeff.bin
I removed the assimilation of amsua_metop-b and tried again. All other sensors worked OK.
Could you check the amsua_metop-b coefficients?
I can reproduce the same issue with the SpcCoeff_Inspect tool:
./SpcCoeff_Inspect ../test_data/3.1.1/fix_REL-3.1.1.1/fix/SpcCoeff/Big_Endian/amsua_metop-b.SpcCoeff.bin
**********************************************************
SpcCoeff_Inspect
Program to display the contents of a CRTM Binary format
R8 SpcCoeff file to stdout.
$Revision$
**********************************************************
SpcCoeff_ReadFile(Binary)(FAILURE) : Error reading channel data. input statement requires too much data, unit 10, file /data/users/bjohnson/CRTM/CRTMv3_REL-3.1.1/build/test/../test_data/3.1.1/fix_REL-3.1.1.1/fix/SpcCoeff/Big_Endian/amsua_metop-b.SpcCoeff.bin
It's likely related to the coefficient version number. I'll dig into it now.
Andrew found this bug in CRTM v2.4 recently, and the fix wasn't propagated into v3. (https://github.com/JCSDA/crtm/pull/43)
I'll work on a PR now.
@emilyhcliu if you can check out the branch: hotfix/btj_fix_version_check_spccoeff_binary_io_REL-3.1.1
-- this should fix the error. Please verify that it doesn't break other things. If it works out, I'll update the coefficient tarball with the updated LE file, and update release/REL-3.1.1 with this fix (I also added it to develop).
@emilyhcliu if you can check out the branch:
hotfix/btj_fix_version_check_spccoeff_binary_io_REL-3.1.1
-- this should fix the error. Please verify that it doesn't break other things. If it works out, I'll update the coefficient tarball with the updated LE file, and update release/REL-3.1.1 with this fix (I also added it to develop).
OK! Working on it and report back here.
@BenjaminTJohnson
I checked out the hotfix/btj_fix_version_check_spccoeff_binary_io_REL-3.1.1
and ran two GDAS experiments with all observation types, big and little endian, respectively
Both runs were completed successfully, and the results (O-F and data passed QC) are comparable to those from the operational system.
@emilyhcliu awesome, thank you for checking!
@emilyhcliu awesome, thank you for checking!
@BenjaminTJohnson
You are welcome!
Are you going to update relaese/REL-3.1.1
with the fixes from the hotfix branch and then create a new tag?
@emilyhcliu yes, I'm working on it now, will let you know when I have a new tag for release/REL-3.1.1
Should be today.
Preliminary validation of TROPICS using CRTMv3.1.1 with the TROPICS coefficients delivery on 08302024
. In this coefficient delivery, the naming convention for TROPICS changed to the following:
tms_tropics-01, tms_tropics-02, tms_tropics-03, ..., tms_tropics-07
Currently, only TROPICS-03
, TROPICS-05
, and TROPICS-06
are in operational status.
So, the validation of TROPICS was performed for these three TROPICS satellites.
Summary:
For example, TROPICS-03 channel 2 simulated BT values look too low (see the first plot below) compared to the same channel from TROPICS-06 (see the second plot below). The emissivity values are comparable. So, something is wrong with the coefficients from TROPICS-03. All other channels from TROPICS-03 have the same problem.
Here is the example
icmask : over ocean
lat lon : 6.91820000000000 348.506310000000
channel no. : 9
emissivity : 0.680391304489999 0.000000000000000E+000
tsim tsim_clr : -4.855339803624857962E+07 2.388710652256545188E+02
total_od : 105.287784254355
tcc : 0.999996376457260
Layer Pressure Temperature Moisture RH Transmittance Layer Optical Depth Cloud Fraction
46 0.112368755452404599E+03 0.192948462139890580E+03 0.352237515743650904E-02 0.118441222522142575E+01 0.984267912974268744E+00 0.627289539468096208E-02 0.100000000000000000E+01
47 0.120159963699790083E+03 0.195010011677196673E+03 0.530941336449387933E-02 0.141851828835153904E+01 0.955737898333287594E+00 0.289904275737285257E-01 0.100000000000000000E+01
48 0.128321931001968267E+03 0.198251431525769362E+03 0.913348737092145764E-02 0.155393938316896718E+01 0.893223071250948353E+00 0.666722649776307352E-01 0.100000000000000000E+01
49 0.136864180694765338E+03 0.201298209494333463E+03 0.130024834898564215E-01 0.146679308211965598E+01 0.808946417522109718E+00 0.976751478296985082E-01 0.100000000000000000E+01
50 0.145795486446628814E+03 0.204544775824695705E+03 0.202593913081536951E-01 0.149171737239978541E+01 0.727175642492371765E+00 0.105028569359558038E+00 0.100000000000000000E+01
51 0.155123746674179529E+03 0.207770956490796891E+03 0.297951400349664650E-01 0.145531473237471753E+01 0.670402782643488404E+00 0.801176104191126115E-01 0.100000000000000000E+01
52 0.164855893895915301E+03 0.211041778168916323E+03 0.431277934862537432E-01 0.140984322386916183E+01 0.655460166734705307E+00 0.222162489188298361E-01 0.100000000000000000E+01
53 0.174997710870320589E+03 0.214335535560490712E+03 0.617697275954307012E-01 0.136529859850888324E+01 0.700777499868205256E+00 -0.658892547995999917E-01 0.100000000000000000E+01
54 0.185553738029519991E+03 0.217665378297667218E+03 0.873332587148814149E-01 0.131748427729100204E+01 0.840514807232376704E+00 -0.179203251470459485E+00 0.100000000000000000E+01
55 0.196527138400002229E+03 0.220969451105517066E+03 0.122380758562382272E+00 0.127934043459363567E+01 0.115732702643507146E+01 -0.315243267929734561E+00 0.100000000000000000E+01
56 0.207919540026004995E+03 0.224301246954333465E+03 0.169436993888227888E+00 0.123871120695182291E+01 0.186194766259390554E+01 -0.468655820703007708E+00 0.100000000000000000E+01
57 0.219730943316417552E+03 0.227568202169135702E+03 0.232739310167697749E+00 0.121250154903814344E+01 0.357072631435203558E+01 -0.641760076684931513E+00 0.100000000000000000E+01
58 0.231959558755227135E+03 0.230779442822262922E+03 0.312895227513714347E+00 0.118077569218216971E+01 0.824802065410952245E+01 -0.825136430680274202E+00 0.100000000000000000E+01
59 0.244601743008196053E+03 0.233913359805495134E+03 0.416553205266905036E+00 0.116064179545471702E+01 0.233501022801095814E+02 -0.102562808483912882E+01 0.100000000000000000E+01
60 0.257651829770476809E+03 0.236939023891545759E+03 0.549843701941331586E+00 0.115420842857638317E+01 0.798369776910072204E+02 -0.121166456058449068E+01 0.100000000000000000E+01
61 0.271101930362848350E+03 0.239860925678088620E+03 0.706732012915749608E+00 0.113866544491397637E+01 0.320754150461042173E+03 -0.137064223259729134E+01 0.100000000000000000E+01
62 0.284941893151870829E+03 0.242713826805646534E+03 0.892381160665631779E+00 0.111860643921549818E+01 0.147397354127264407E+04 -0.150305961273278887E+01 0.100000000000000000E+01
63 0.299159348057764532E+03 0.245432137711676546E+03 0.110551000855024251E+01 0.109928890755727826E+01 0.750818528049870474E+04 -0.160456485087540557E+01 0.100000000000000000E+01
64 0.313739716647835962E+03 0.248016029335115036E+03 0.134767134920098175E+01 0.108282948046497940E+01 0.408832162437083607E+05 -0.167029736787144123E+01 0.100000000000000000E+01
65 0.328666163484191941E+03 0.250522622974734190E+03 0.162681992781712004E+01 0.106844222969337732E+01 0.227930660959769506E+06 -0.169355326955701480E+01 0.100000000000000000E+01
66 0.343919641283895544E+03 0.252847315002393259E+03 0.192513184263021642E+01 0.105325187653739039E+01 0.124592333921261318E+07 -0.167410654382820545E+01 0.100000000000000000E+01
67 0.359478944240781971E+03 0.255082476083769620E+03 0.225297347959660366E+01 0.102432736104646893E+01 0.641926583191901911E+07 -0.161579550906801606E+01 0.100000000000000000E+01
68 0.375320792920089048E+03 0.257197508751600026E+03 0.260553149793525041E+01 0.999169290844158398E+00 0.298638691781954020E+08 -0.151518569345517595E+01 0.100000000000000000E+01
69 0.391419905135327326E+03 0.259146337272336382E+03 0.296857690737864344E+01 0.982301254118353118E+00 0.122443059165241137E+09 -0.139065806795419999E+01 0.100000000000000000E+01
70 0.407749123703416387E+03 0.261018926395882829E+03 0.335751244740984900E+01 0.970500429095716211E+00 0.440509229986245334E+09 -0.126183075680462564E+01 0.100000000000000000E+01
71 0.424279601209808447E+03 0.262784035482675449E+03 0.376061611836828780E+01 0.963539118262376371E+00 0.138903963087451196E+10 -0.113188245424781542E+01 0.100000000000000000E+01
72 0.440980972861858959E+03 0.264379346083455857E+03 0.415567586969260727E+01 0.961722914129143791E+00 0.387680629636699247E+10 -0.101160417383636014E+01 0.100000000000000000E+01
73 0.457821591633499452E+03 0.265917377094614949E+03 0.456830716609579035E+01 0.962129730297707875E+00 0.980117483328488922E+10 -0.914121345287031706E+00 0.100000000000000000E+01
74 0.474768715546678322E+03 0.267381158702413131E+03 0.498277929800677377E+01 0.963421132320651519E+00 0.226938099882273064E+11 -0.827487755819400594E+00 0.100000000000000000E+01
75 0.491788758866018497E+03 0.268787649563512673E+03 0.539948351817316574E+01 0.964951457884979047E+00 0.484555230262334442E+11 -0.747620024871503475E+00 0.100000000000000000E+01
76 0.508847589934717348E+03 0.270124119805240525E+03 0.580529980387877576E+01 0.966006822778305962E+00 0.965056665614517212E+11 -0.679024523305099326E+00 0.100000000000000000E+01
77 0.525910778295324690E+03 0.271441168814241792E+03 0.621372992878328478E+01 0.965764804036856050E+00 0.177285967093166321E+12 -0.599396040191106816E+00 0.100000000000000000E+01
78 0.542943847917869107E+03 0.272500489526112347E+03 0.659475224143999927E+01 0.977993545069821613E+00 0.188201769227405396E+12 -0.588892972646105420E-01 0.100000000000000000E+01
79 0.559912599245759338E+03 0.273594377667511935E+03 0.701007486679936953E+01 0.989286221853408532E+00 0.119403862496893616E+12 0.448444482369592834E+00 0.100000000000000000E+01
80 0.576783352437270082E+03 0.274645003015366456E+03 0.742502044782967197E+01 0.999626857295710547E+00 0.362602205045220261E+11 0.117461132289693948E+01 0.100000000000000000E+01
81 0.593523209689944224E+03 0.276012203908990045E+03 0.794042667483109277E+01 0.997570226589861275E+00 0.719473793743875408E+10 0.159405793363552140E+01 0.100000000000000000E+01
82 0.610100327012707680E+03 0.277322812592357423E+03 0.843373930293878615E+01 0.992483828266428558E+00 0.766290570243414879E+09 0.220726208631862253E+01 0.100000000000000000E+01
83 0.626484120454552794E+03 0.278400072353277892E+03 0.884927214360775416E+01 0.991187505647120548E+00 0.104841582215629935E+09 0.196043912343032334E+01 0.100000000000000000E+01
84 0.642645492819719721E+03 0.279411225869916848E+03 0.925604645008835547E+01 0.990952632778963505E+00 0.331018831988864318E+08 0.113624250691173234E+01 0.100000000000000000E+01
85 0.658557013433576685E+03 0.280364930560214930E+03 0.965222650284527894E+01 0.991183188847337804E+00 0.144310428791106530E+08 0.818241576263193826E+00 0.100000000000000000E+01
86 0.674193105364881717E+03 0.281273565598070547E+03 0.100354896321537375E+02 0.991026192619349988E+00 0.639545296894282289E+07 0.802064020741179107E+00 0.100000000000000000E+01
87 0.689530126870964750E+03 0.282138423751850041E+03 0.104097833948930774E+02 0.990980367926629580E+00 0.288836097268483089E+07 0.783440082775412217E+00 0.100000000000000000E+01
88 0.704546511519773503E+03 0.282958158395488908E+03 0.107707099066923035E+02 0.990899005258641274E+00 0.117364668070812267E+07 0.887592255667218666E+00 0.100000000000000000E+01
89 0.719222908020927662E+03 0.283730880754588952E+03 0.111202103088698081E+02 0.991250772613729403E+00 0.396672082618789515E+06 0.106912487220368413E+01 0.100000000000000000E+01
90 0.733542144544168195E+03 0.284470647772360280E+03 0.114563409839619972E+02 0.991059818854393759E+00 0.177472661241370399E+06 0.792699969594861331E+00 0.100000000000000000E+01
91 0.747489241315466984E+03 0.285170671444095319E+03 0.117814221692832835E+02 0.991114507536498324E+00 0.881529164654989436E+05 0.689657199207583504E+00 0.100000000000000000E+01
92 0.761051496050116725E+03 0.285838581413680117E+03 0.120966766878583236E+02 0.991110248754739631E+00 0.449884899505943322E+05 0.662970234788192170E+00 0.100000000000000000E+01
93 0.774218370525351247E+03 0.286473941211435260E+03 0.124007739670579546E+02 0.991078591638424267E+00 0.236033741603706294E+05 0.635719471090700927E+00 0.100000000000000000E+01
94 0.786981495816953611E+03 0.287078960309659806E+03 0.126957980621973494E+02 0.991111195500771647E+00 0.127181561745598683E+05 0.609445807667855277E+00 0.100000000000000000E+01
95 0.799334589345327231E+03 0.287652195558099606E+03 0.129783461492503776E+02 0.991135714796757439E+00 0.703817243637718821E+04 0.583153312081281183E+00 0.100000000000000000E+01
96 0.811273330505903914E+03 0.288192564407905479E+03 0.132501704873896085E+02 0.991421151924577804E+00 0.399303044425244752E+04 0.558628033191450046E+00 0.100000000000000000E+01
97 0.822795341343545374E+03 0.288715219637135419E+03 0.135166657223443405E+02 0.991462921341760506E+00 0.174513164622633712E+04 0.815789364840150411E+00 0.100000000000000000E+01
98 0.833900000154085774E+03 0.289213233584794921E+03 0.137759469649759545E+02 0.991622797558799229E+00 0.589649181915120693E+02 0.333881178169608983E+01 0.100000000000000000E+01
99 0.844588385462345514E+03 0.289691350356601617E+03 0.140288840976259674E+02 0.991704909301065363E+00 0.236142827774331376E+01 0.317129515605179968E+01 0.100000000000000000E+01
100 0.854863083660557209E+03 0.290149786650059866E+03 0.142781695318257142E+02 0.991936822485589231E+00 0.261452925285869109E-01 0.443843963807860575E+01 0.100000000000000000E+01
101 0.864728095870725610E+03 0.290585845774668655E+03 0.145199061429880398E+02 0.992255683627020768E+00 0.205020777698625806E-03 0.477842758757468911E+01 0.100000000000000000E+01
Layer Pressure Temperature Moisture RH Transmittance. Layer Optical Depth Cloud Fraction
@BenjaminTJohnson and @emilyhcliu we are also seeing something off with the TROPICS-03 coefficients this is using :
crtm/fix_REL-3.1.1.2/fix/TauCoeff/ODPS/Little_Endian/tms_tropics-0*
crtm/fix_REL-3.1.1.2/fix/SpcCoeff/Little_Endian/tms_tropics-0*
This is odd. tms_tropics-03 is quite similar to tms_tropics-05, see below. Can you send me the shasum of the SpcCoeff and TauCoeff files that you're using, just to make sure and to also verify that it's not a taucoeff issue?
There should be almost no difference between the two for the same profile (based on SpcCoeff).
[s4-submit] /data/users/bjohnson/CRTM/fix_directories> diff -y a c
**********************************************************
SpcCoeff_Inspect
SpcCoeff_Inspect
Program to display the contents of a CRTM Binary/netCDF Program
to display the contents of a CRTM Binary/netCDF
format R8 SpcCoeff file to stdout. format R8 SpcCoeff file to
stdout.
$Revision$ $Revision$
**********************************************************
SpcCoeff_ReadFile(Binary)(INFORMATION) : FILE: ./fix_REL-3.1.1.2/fix/SpcCoeff/Little_En | SpcCoeff_ReadFile(Binary)(INFORMATION) : FILE: ./fix_REL-3.1.1.2/fix/SpcCoeff/Little_Endian/
SpcCoeff RELEASE.VERSION: 8.04 SpcCoeff RELEASE.VERSION: 8.04
N_CHANNELS=12 N_CHANNELS=12
SpcCoeff OBJECT SpcCoeff OBJECT
Release.Version : 8.4 Release.Version : 8.4
n_Channels : 12 n_Channels : 12
Sensor_Id : tms_tropics-03 | Sensor_Id : tms_tropics-05
WMO_Satellite_ID : 1023 WMO_Satellite_ID : 1023
WMO_Sensor_ID : 2047 WMO_Sensor_ID : 2047
Sensor_Type : Microwave Sensor_Type : Microwave
Sensor_Channel : Sensor_Channel :
1 2 3 4 5 6 7 8 9 10 1 2
3 4 5 6 7 8 9 10
11 12 11 12
Polarization : Polarization :
Channel 1: Mixed polarization with constant mixing angle Channel
1: Mixed polarization with constant mixing angle
Channel 2: Mixed polarization with constant mixing angle Channel
2: Mixed polarization with constant mixing angle
Channel 3: Mixed polarization with constant mixing angle Channel
3: Mixed polarization with constant mixing angle
Channel 4: Mixed polarization with constant mixing angle Channel
4: Mixed polarization with constant mixing angle
Channel 5: Mixed polarization with constant mixing angle Channel
5: Mixed polarization with constant mixing angle
Channel 6: Mixed polarization with constant mixing angle Channel
6: Mixed polarization with constant mixing angle
Channel 7: Mixed polarization with constant mixing angle Channel
7: Mixed polarization with constant mixing angle
Channel 8: Mixed polarization with constant mixing angle Channel
8: Mixed polarization with constant mixing angle
Channel 9: Mixed polarization with constant mixing angle Channel
9: Mixed polarization with constant mixing angle
Channel 10: Mixed polarization with constant mixing angle Channel
10: Mixed polarization with constant mixing angle
Channel 11: Mixed polarization with constant mixing angle Channel
11: Mixed polarization with constant mixing angle
Channel 12: Mixed polarization with constant mixing angle Channel
12: Mixed polarization with constant mixing angle
Fixed Polarization Angle: Fixed Polarization Angle:
Channel 1: Channel 1:
1.600000000000000E+01 1.600000000000000E+01
Channel 2: Channel 2:
1.400000000000000E+01 1.400000000000000E+01
Channel 3: Channel 3:
1.400000000000000E+01 1.400000000000000E+01
Channel 4: Channel 4:
1.400000000000000E+01 1.400000000000000E+01
Channel 5: Channel 5:
1.400000000000000E+01 1.400000000000000E+01
Channel 6: Channel 6:
1.400000000000000E+01 1.400000000000000E+01
Channel 7: Channel 7:
1.400000000000000E+01 1.400000000000000E+01
Channel 8: Channel 8:
1.400000000000000E+01 1.400000000000000E+01
Channel 9: Channel 9:
-7.600000000000000E+01 -7.600000000000000E+01
Channel 10: Channel 10:
-7.600000000000000E+01 -7.600000000000000E+01
Channel 11: Channel 11:
-7.600000000000000E+01 -7.600000000000000E+01
Channel 12: Channel 12:
-7.600000000000000E+01 -7.600000000000000E+01
Channel_Flag : Channel_Flag :
00000000000000000000000000000000 00000000000000000000000000000000 00000000000000000000000000 00000000000000000000000000000000 00000000000000000000000000000000 00000000000000000000000000
00000000000000000000000000000000 00000000000000000000000000000000 00000000000000000000000000 00000000000000000000000000000000 00000000000000000000000000000000 00000000000000000000000000
00000000000000000000000000000000 00000000000000000000000000000000 00000000000000000000000000 00000000000000000000000000000000 00000000000000000000000000000000 00000000000000000000000000
00000000000000000000000000000000 00000000000000000000000000000000 00000000000000000000000000 00000000000000000000000000000000 00000000000000000000000000000000 00000000000000000000000000
Frequency : Frequency :
9.117958190941538E+01 1.144591690786891E+02 1.159068843575291E+02 1.165735520839200E+02 | 9.125624920399511E+01 1.144568755699984E+02 1.158994320011776E+02 1.165606594043212E+02
1.177481979493747E+02 1.181869948561453E+02 1.185424868634187E+02 1.843164511080561E+02 | 1.177912933368352E+02 1.181958902259432E+02 1.185727171630761E+02 1.841724972901750E+02
1.901800681981482E+02 2.046003163670611E+02 | 1.902196339477300E+02 2.046921715197209E+02
Wavenumber : Wavenumber :
3.041423474015994E+00 3.817946917086523E+00 3.866237500795603E+00 3.888475142490744E+00 | 3.043980819690772E+00 3.817870413871398E+00 3.865988916945257E+00 3.888045088990239E+00
3.927657111019608E+00 3.942293800337875E+00 3.954151737313508E+00 6.148135024399317E+00 | 3.929094618412155E+00 3.942590517935685E+00 3.955160111568789E+00 6.143333241898133E+00
6.343724237323817E+00 6.824731940623439E+00 | 6.345044008669824E+00 6.827795898712065E+00
Planck_C1 : Planck_C1 :
3.350873403960535E-04 6.628526509168412E-04 6.883240226430110E-04 7.002696768976313E-04 | 3.359333142432389E-04 6.628128054103948E-04 6.881912616049909E-04 7.000373595166093E-04
7.216523645068532E-04 7.297503315581682E-04 7.363551571822456E-04 2.767945472133319E-03 | 7.224450205125531E-04 7.299151184111399E-04 7.369186491805282E-04 2.761465119508226E-03
3.040606555227588E-03 3.786030722414100E-03 | 3.042504686438767E-03 3.791132218672646E-03
Planck_C2 : Planck_C2 :
4.375924807215824E+00 5.493167515095018E+00 5.562646811553681E+00 5.594641779955517E+00 | 4.379604252868327E+00 5.493057444162853E+00 5.562289155263232E+00 5.594023029624507E+00
5.651015826366953E+00 5.672074732642423E+00 5.689135638832365E+00 8.845784482572176E+00 | 5.653084076419385E+00 5.672501642577223E+00 5.690586462749632E+00 8.838875796772484E+00
9.127193400525323E+00 9.819255377199491E+00 | 9.129092254868423E+00 9.823663724252386E+00
Band_C1 : Band_C1 :
-1.148129809195098E-03 -5.065043234253608E-05 -3.226826228797108E-05 -1.953483788952326E-05 | -1.800271585864266E-03 -5.806981917544363E-05 -3.199554063826326E-05 -1.716501611781496E-05
-1.164127482411459E-05 -9.068161148206855E-06 -6.478811627630421E-06 -1.583168059653417E-04 | -1.267164330442938E-05 -1.048913864565293E-05 -5.685016532197551E-06 -2.124134801988475E-04
-1.960989096971844E-04 -1.475340457375296E-04 | -2.051855006470760E-04 -1.685418583292631E-04
Band_C2 : Band_C2 :
1.000177010499153E+00 1.000006261696347E+00 1.000003940572505E+00 1.000002372039655E+00 | 1.000277432682579E+00 1.000007179284426E+00 1.000003907446076E+00 1.000002084622262E+00
1.000001399764893E+00 1.000001086364264E+00 1.000000773893260E+00 1.000012287908244E+00 | 1.000001523258319E+00 1.000001256479487E+00 1.000000678963498E+00 1.000016501647486E+00
1.000014766401811E+00 1.000010350613367E+00 | 1.000015448911445E+00 1.000011819953787E+00
Cosmic_Background_Radiance : Cosmic_Background_Radiance :
8.416780431845172E-05 1.018930783062087E-04 1.027475345181134E-04 1.031309987734977E-04 | 8.423790427023252E-05 1.018917010493619E-04 1.027432122725136E-04 1.031236428378649E-04
1.037912843939782E-04 1.040329088671221E-04 1.042266555425931E-04 1.121429232673956E-04 | 1.038151361071290E-04 1.040377788319012E-04 1.042430485334745E-04 1.121758870341300E-04
1.106642696191594E-04 1.060269659290278E-04 | 1.106534212858444E-04 1.059934369781432E-04
Solar_Irradiance : Solar_Irradiance :
0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00
On Wed, Sep 11, 2024 at 2:48 PM Benjamin Ruston @.***> wrote:
@BenjaminTJohnson https://github.com/BenjaminTJohnson and @emilyhcliu https://github.com/emilyhcliu we are also seeing something off with the TROPICS-03 coefficients this is using : crtm/fix_REL-3.1.1.2/fix/TauCoeff/ODPS/Little_Endian/tms_tropics-0 crtm/fix_REL-3.1.1.2/fix/SpcCoeff/Little_Endian/tms_tropics-0
image.png (view on web) https://github.com/user-attachments/assets/a08f2d14-cbca-443a-9500-efa082a1992c
— Reply to this email directly, view it on GitHub https://github.com/JCSDA/CRTMv3/issues/167#issuecomment-2344427895, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7B5FS3U3VVBVBLKVSJJKLZWCF7DAVCNFSM6AAAAABMYIV2S2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNBUGQZDOOBZGU . You are receiving this because you were mentioned.Message ID: @.***>
@BenjaminTJohnson, here is a list of the shasums for tms_tropics-03
& -05
(little endian only):
3a6f136c2c5aa7bcde37365af44e154e4b5b2037 ./fix_REL-3.1.1.2/fix/SpcCoeff/Little_Endian/tms_tropics-05.SpcCoeff.bin
4b5e32105fb5ba127b3907b3021c4b25cf4077e8 ./fix_REL-3.1.1.2/fix/SpcCoeff/Little_Endian/tms_tropics-03.SpcCoeff.bin
cd8d5cffdb9b2fd2d97fc047f5f973a5e96dd3fa ./fix_REL-3.1.1.2/fix/TauCoeff/ODPS/Little_Endian/tms_tropics-03.TauCoeff.bin
2d8b6e91c048caf23ac9c5aa4b9180cbd5c16614 ./fix_REL-3.1.1.2/fix/TauCoeff/ODPS/Little_Endian/tms_tropics-05.TauCoeff.bin
I'm also seeing the TB calculations are different when using the netcdf and binary files for tms_tropics-03 only. I will fix this today and provide an update. All of the other coefficients look fine. The problem isn't immediately obvious, so it might take some poking around.
@ADCollard @emilyhcliu @fabiolrdiniz @BenjaminRuston
I recreated all of the tms_tropics-03 coefficients -- here's a link to the updated delivery folder, a link to the tropics-only tarball on the ftp (updated only tms_tropics-03.SpcCoeff and tms_tropics-03.TauCoeff)
https://bin.ssec.wisc.edu/pub/s4/CRTM/tropics_09122024_delivery.tgz
https://drive.google.com/drive/folders/185UYpr_PhwfckKHtJtWBexlfgo5L41sm?usp=sharing
The next tarball release of CRTM coefficients fix_REL-3.1.1.3
will contain the fixed files, no ETA on that yet.
@BenjaminTJohnson we've just run a quick test with your new files from the tropics_09122024_delivery.tgz
tarball with these shasums:
215805266650adb666740dd5c2e88603e849ff04 tms_tropics-03.TauCoeff.bin
29c6ac573b1e5955b74c30e94cc3087722b17634 tms_tropics-03.SpcCoeff.bin
2d8b6e91c048caf23ac9c5aa4b9180cbd5c16614 tms_tropics-05.TauCoeff.bin
3a6f136c2c5aa7bcde37365af44e154e4b5b2037 tms_tropics-05.SpcCoeff.bin
8cec0aad9e90fdeb82142790e2b04d05336d1ffb tms_tropics-06.TauCoeff.bin
d0b14718150d21717cd4c123e268d90e0ea9af7d tms_tropics-06.SpcCoeff.bin
tms_tropics-03 nobs= 4177524 Min=52.0596, Max=330.508, RMS=200.854
tms_tropics-05 nobs= 4864116 Min=62.314, Max=303.619, RMS=255.359
tms_tropics-06 nobs= 5049072 Min=64.1236, Max=304.206, RMS=255.965
tms_tropics-03
):tms_tropics-03 nobs= 4177524 Min=53.963, Max=323.349, RMS=253.674
tms_tropics-05 nobs= 4864116 Min=62.314, Max=303.619, RMS=255.359
tms_tropics-06 nobs= 5049072 Min=64.1236, Max=304.206, RMS=255.965
And following @BenjaminRuston's https://github.com/JCSDA/CRTMv3/issues/167#issuecomment-2344427895 figure:
@fabiolrdiniz this looks to be "better". OMB values would be helpful for overall quality of the coefficients. RMS=253.674
for the new tms_tropics-03
coefficient is now consistent with the other sensors.
Observation
Observation
@fabiolrdiniz @BenjaminRuston @BenjaminTJohnson The following table is from the previous comment:
tms_tropics-03 nobs= 4177524 Min=53.963, Max=323.349, RMS=253.674
tms_tropics-05 nobs= 4864116 Min=62.314, Max=303.619, RMS=255.359
tms_tropics-06 nobs= 5049072 Min=64.1236, Max=304.206, RMS=255.965
The minimum values of HofX are too low. (~50K) Could you identify the channel?
Thanks.
I agree that you'll never see TBs that low in reality or in correct simulations. 90K is really the lowest I've ever seen occurring naturally.
My guess is that this is an ice cloud, with too much ice, too high in the atmosphere. I'll leave it to Fabio to identify which channel produces these super low TBs, but my guess is the 205 GHz channel (12)
Here's an example of simulated TBs that get down to 90K in hurricanes at 183+/-7 (3rd row 3rd column is the obvious example). [image: image.png]
I agree that you'll never see TBs that low in reality or in correct simulations. 90K is really the lowest I've ever seen occurring naturally. My guess is that this is an ice cloud, with too much ice, too high in the atmosphere. I'll leave it to Fabio to identify which channel produces these super low TBs, but my guess is the 205 GHz channel (12) Here's an example of simulated TBs that get down to 90K in hurricanes at 183+/-7 (3rd row 3rd column is the obvious example). [image: image.png]
@BenjaminTJohnson The image does not show. I agree. Channel 12 could depress the BT a lot more than the other channels due to ice particles.
The minimum values of HofX are too low. (~50K) Could you identify the channel?
@emilyhcliu it's from channel 12. Please, see below a summary for each of the platforms:
tms_tropics-03
: ObsValue hofx
min max min max
variable channel_number expt_name refTime
brightnessTemperature 1 Skylab 2024-07-21 172.913635 303.001495 61.776646 323.348907
2 Skylab 2024-07-21 158.448578 302.457031 68.563171 309.759827
3 Skylab 2024-07-21 167.767960 294.705841 81.259605 299.327820
4 Skylab 2024-07-21 172.447098 290.377594 94.906532 289.367462
5 Skylab 2024-07-21 189.056198 275.273651 116.515747 275.196838
6 Skylab 2024-07-21 192.379044 253.519470 158.157806 250.134247
7 Skylab 2024-07-21 194.303268 232.227081 193.123016 226.918442
8 Skylab 2024-07-21 194.990021 228.500717 209.523651 227.921768
9 Skylab 2024-07-21 139.583603 272.068359 91.893646 273.055817
10 Skylab 2024-07-21 128.730835 283.815002 63.313141 283.140350
11 Skylab 2024-07-21 124.127464 287.819489 55.737331 287.912476
12 Skylab 2024-07-21 105.751259 299.623657 53.962959 300.024292
tms_tropics-05
: ObsValue hofx
min max min max
variable channel_number expt_name refTime
brightnessTemperature 1 Skylab 2024-07-21 168.433609 294.613373 67.484467 300.656982
2 Skylab 2024-07-21 NaN NaN 75.181709 297.418762
3 Skylab 2024-07-21 162.248917 289.225006 85.266518 292.059021
4 Skylab 2024-07-21 155.938293 291.721008 96.014412 285.808777
5 Skylab 2024-07-21 173.629395 276.906006 115.257484 274.202576
6 Skylab 2024-07-21 176.872986 254.874588 153.549255 251.098480
7 Skylab 2024-07-21 168.738129 235.059357 187.347626 227.579742
8 Skylab 2024-07-21 170.504486 230.131027 209.764999 224.949203
9 Skylab 2024-07-21 145.359970 274.139282 95.900795 278.354095
10 Skylab 2024-07-21 131.273499 287.919220 69.265396 290.216675
11 Skylab 2024-07-21 126.429901 297.038177 63.696941 298.992371
12 Skylab 2024-07-21 107.792717 303.699432 62.313972 303.619141
tms_tropics-06
: ObsValue hofx
min max min max
variable channel_number expt_name refTime
brightnessTemperature 1 Skylab 2024-07-21 156.818680 293.861145 72.122696 301.469208
2 Skylab 2024-07-21 140.654129 292.907379 78.676331 298.090363
3 Skylab 2024-07-21 150.897171 288.585205 89.086189 292.420074
4 Skylab 2024-07-21 147.426880 286.750214 100.768776 285.505188
5 Skylab 2024-07-21 165.699173 276.232178 119.331680 274.118317
6 Skylab 2024-07-21 185.795166 255.912277 157.085403 250.934937
7 Skylab 2024-07-21 196.067245 230.805954 188.515778 227.197754
8 Skylab 2024-07-21 204.111374 227.799042 206.749329 224.116791
9 Skylab 2024-07-21 142.847015 275.644623 96.426811 279.177063
10 Skylab 2024-07-21 124.919464 289.249481 71.504639 290.128632
11 Skylab 2024-07-21 122.377251 297.351166 66.100174 298.999115
12 Skylab 2024-07-21 100.142387 306.118958 64.123611 304.205597
Yeah, we're producing too much cloud scattering to be able to generate that low of a TB. @gthompsnJCSDA also sees this in visible / IR images, where we have too much brightness / albedo too high. This suggests that the cloud particle sizes (effective radius) or number concentrations are off. Could also be cloud height errors. Could also be the CloudCoeff LUT.
@fabiolrdiniz Thanks for the summary.
It looks like the minimum values of HofX for G-band are low, but they are also low for channels 1-3.
What is the background model? GFS or other?
What is the background model? GFS or other?
It's GFS c768 grabbed from the NOAA's AWS bucket.
What is the background model? GFS or other?
It's GFS c768 grabbed from the NOAA's AWS bucket.
So, we are using the same profiles.
What is your Total Cloud Cover?
Are you assuming overcast?
Here are the background fields
So, we are using the same profiles.
Somewhat yes... different pre-processings since you are relying on a BUFR file for GSI and we are converting NASA's file into IODA. And GSI might have some QC being applied too, right?
What is your Total Cloud Cover? Are you assuming overcast?
We are using Cloud_Fraction: 1.0
in our YAMLs. We are not using it from the model at this point.
Another point of difference between our runs is that you might have access to backgrounds throughout the window via GSI (due to the 4D-flavor settings), while we are seeing numbers referred to an approximated 3D case.
On another topic, @BenjaminRuston asked me to share with you the spaghetti plots previously:
The ice signals are weaker in my simulation. The location of the signal is right, but much weaker.
But, we are using the same GFS backend. I am wondering why.
One thing to note that the current GFS operational system is using GFDL cloud physics not Thompson
So, I am using GFDL cloud table in my simulation.
I am using GFDL cloud table in my simulation.
I think this is the easiest thing for us to change. We're still using the default CRTM "CloudCoeff.bin", which is most likely a strong contributor here.
One thing to note that the current GFS operational system is using GFDL cloud physics not Thompson So, I am using GFDL cloud table in my simulation.
For the effective radius method, as far as I remember, @gthompsnJCSDA set fv3-jedi to use GFDL as default. It's worth checking the code though.
@fabiolrdiniz
./src/ufo/operators/crtm/ufo_crtm_utils_mod.F90: conf%CloudCoeff_File = "CloudCoeff.bin"
change to
first we want to try:
./test_data/3.1.1/fix_REL-3.1.1.2/fix/CloudCoeff/Little_Endian/cloudcoeff_GFDLFV3.bin
then we want to try:
./test_data/3.1.1/fix_REL-3.1.1.2/fix/CloudCoeff/Little_Endian/CloudCoeff.GFDLFV3.-109z-1.bin
probably need to make a change in a CMakeLists.txt file somewhere to find the right file? I'm not that well versed with how UFO is picking up the coefficient files.
So, we are using the same profiles.
Somewhat yes... different pre-processings since you are relying on a BUFR file for GSI and we are converting NASA's file into IODA. And GSI might have some QC being applied too, right?
What is your Total Cloud Cover? Are you assuming overcast?
We are using
Cloud_Fraction: 1.0
in our YAMLs. We are not using it from the model at this point.Another point of difference between our runs is that you might have access to backgrounds throughout the window via GSI (due to the 4D-flavor settings), while we are seeing numbers referred to an approximated 3D case.
On another topic, @BenjaminRuston asked me to share with you the spaghetti plots previously:
@fabiolrdiniz, thanks for sharing more information. It is useful.
The Channel One scan bias is significant in general and at larger scan positions. At the nadir for TROPICS-06 and TROPICS-05, there is an offset of 5K.
Are the samples for the statistics from clear-sky ones?
For the effective radius method, as far as I remember, @gthompsnJCSDA set fv3-jedi to use GFDL as default. It's worth checking the code though.
Confirmed. Please, see: https://github.com/JCSDA-internal/fv3-jedi/blob/72cb04e5a4338a6104b75c891db62ea5b0c9274d/src/fv3jedi/VariableChange/Model2GeoVaLs/fv3jedi_vc_model2geovals_mod.f90#L65
@fabiolrdiniz You mentioned the following:
Somewhat yes... different pre-processings since you are relying on a BUFR file for GSI and we are converting NASA's file into IODA. And GSI might have some QC being applied too, right?
What is NASA's file? Is it BUFR or HDF/NetCDF? We remap all ATMS channels to a common 3x3 FOV size using a utility from AAPP software.
One thing to note that the current GFS operational system is using GFDL cloud physics not Thompson So, I am using GFDL cloud table in my simulation.
For the effective radius method, as far as I remember, @gthompsnJCSDA set fv3-jedi to use GFDL as default. It's worth checking the code though.
Got it. Thanks.
@fabiolrdiniz Could you point to the skylab monitoring webpage with the experiment name and the user name to see the plots for TROPICS?
What is NASA's file? Is it BUFR or HDF/NetCDF?
It's an HDF/NetCDF from NASA (please see https://disc2.gesdisc.eosdis.nasa.gov/data/TROPICS_L1B).
We remap all ATMS channels to a common 3x3 FOV size using a utility from AAPP software.
We are not remapping anything on our side yet.
@fabiolrdiniz Could you point to the skylab monitoring webpage with the experiment name and the user name to see the plots for TROPICS?
Sure. We are testing under the username fabiolrdiniz
and experiment ID 487b11
. However, I wouldn't rely much on the plots over there because I'm running over the same case multiple times. So the plots will change from time to time depending on the tests we are doing.
What is NASA's file? Is it BUFR or HDF/NetCDF?
It's an HDF/NetCDF from NASA (please see https://disc2.gesdisc.eosdis.nasa.gov/data/TROPICS_L1B).
We remap all ATMS channels to a common 3x3 FOV size using a utility from AAPP software.
We are not remapping anything on our side yet.
I am sorry. The TROPICS information I gave you was wrong.
We are receiving TROPICS BUFR files from NESDIS. These BUFR files are converted from NASA's HDF.
We do not remap channels to a common FOV size for TROPICS.
So, we should be using the same information for input.
Sorry about the misunderstanding.
All good! Thanks for checking. That's good information to know.
Validation of CRTMv3.1.1 with GFS background fields from the current operational system (gfs.v16.3)
GSI with CRTMv3.1.1
vs.GSI with CRTMv2.4.0.1
(used in gfs.v16.3)Issues found will be documented here. Tagging @ADCollard for awareness.