IASI Obs Validation

[ADCollard]

Need to validate observations from IASI and evaluate QC procedures in the IASI YAML file, as a IASI YAML file already exists in the GDASApp.

This issue will be used to track progress on validation of radiances in UFO for GDAS in four stages:

• [x] HofX computation using GSI produced GeoVaLs
• [x] QC count checks (and YAML changes) using GSI produced GeoVaLs
• [x] Obs errors using GSI produced GeoVaLs
• [ ] HofX computation in FV3-JEDI

[ADCollard]

So finally we have some HofX differences.

Mean differences are tiny (~1.e-5K) with min/max differences at the 0.01K level, and appear to be mostly related to surface channels.

[ADCollard]

The errant point(s) appears to be on the southern coast of Alaska (look closely!)

[ADCollard]

After a number of (mostly self-inflicted) issues we finally have a plot of differences in QC. It looks like the differences could be a cloud detection issue.

[ADCollard]

So I found the bug in the YAML and now the stats are identical. On to obs errors....

[ADCollard]

Obs errors also appear to be identical. Well done GMAO!

[ADCollard]

Returning to this, post IODA conventions.

We are also using different test files were passive channels are not QC's out.

The HofX continues to align well:

The QC is more problematic. The active channels align perfectly, but the issue seems to be for the active channels from the GSI where passive QC is not correct (I believe this is the same issue as flagged in GSI Issue 528

If we confine to just the passive channels, the JEDI and GSI agree perfectly (and the final obs errors align for all channels - except for one ob in the shortwave band):

[ADCollard]

That was metop-b. Metop-A is worse. There is a major difference for channel 5480:

Looking in detail at the QC flags doesn't make a lot of sense:

In [14]: df.iloc[df.index.get_level_values('Channel') == 5480.0]['PreQC'].value_counts()
Out[14]: 
-7     563   (Cloud)
-51    383  (wavenumber check)
 0     238
-53     88  (emissivity check)
-3       1    (Gross check)
Name: PreQC, dtype: int64

In [6]: df.iloc[df.index.get_level_values('Channel') == 5480.0]['EffectiveQC'].value_counts()
Out[6]: 
12    873
0     399
19      1

Firstly, it appears all channels in the input files are passive. Secondly we would expect that channel to have about the same QC counts as the main window channels, so neither looks right.

[ADCollard]

Upon further investigation, the difference is because the ifail_2000_qc (flag 51) does not actually cause the ob to be rejected:

! Reduce weight given to obs for shortwave ir if
! solar zenith angle tiny_r_kind
  irday = 1
  if (pangs <= 89.0_r_kind .and. frac_sea > zero) then
!    QC2 in statsrad
     if(luse)aivals(9,is) = aivals(9,is) + one
     do i=1,nchanl
        if(wavenumber(i) > r2000)then
           if(wavenumber(i) > r2400)then
              varinv(i)=zero
              varinv_use(i)=zero
              if(id_qc(i) == igood_qc)id_qc(i)=ifail_2400_qc
              irday(i) = 0
           else
              tmp=one-(wavenumber(i)-r2000)*ptau5(1,i)&
                 *max(zero,cos(pangs*deg2rad))*oneover400
              varinv(i)=tmp*varinv(i)
              varinv_use(i)=tmp*varinv_use(i)
              if(id_qc(i) == igood_qc)id_qc(i)=ifail_2000_qc
           end if
        end if
     end do
  endif

If we look at the GSIFinalObsError flag it matches with the number rejected by JEDI.

In [5]: df.iloc[df.index.get_level_values('Channel') == 5480.0]['GsiFinalObsError'].value_counts()
Out[5]: 
-3.368795e+38    874
 1.980007e+00      3
 1.980111e+00      2
 1.980018e+00      2
 1.980003e+00      2
                ... 
 1.980172e+00      1
 1.980130e+00      1
 1.980087e+00      1
 1.980190e+00      1
 1.981521e+00      1
Name: GsiFinalObsError, Length: 391, dtype: int64