Preparation for turning on CrIS NPP radiances and include IASI Metop-C in operational GFS system

[emilyhcliu]

Timeline of Issus and Actions related to CrIS NPP

• 20210521 around 11:30 EDT: CrIS NPP began to show side-2 LW band problem
• 20210521 18Z: NESDIS turned off the CrIS NPP data stream (no CrIS NPP data at all)
• 20210528 12Z: NCO implemented satinfo change (set all CrIS NPP channels to -1)
• 20210602 12Z: NESDIS began to distribute side-2 MW+SW (no CrIS NPP LW data at all)
• 20210712: NESDIS began the process of switching CrIS NPP from side-2 to side-1
• 20210714: NESDIS completed the switch of CrIS NPP to side-1 and began the evaluation process
• 20210721: NESDIS completed the evaluation and held a review meeting (slides).
  The side-2 LW+SW data is in good shape. Data quality is comparable to those from side-1 before. After the side switch occurs, SNPP CrIS data will only be available from CLASS and GRAVITE. These data will be suspended from NDE/PDA until cleared by the calibration/validation science team.
• NOW: we need to prepare for using CrIS NPP again.

Official messages from NESDIS regarding the status CrIS NPP data

Preparation for CrIS NPP (switch back to side-1 for LW + SW)

Plan-A
The plan to cope with the scenario that we have to re-estimate the
correlated observation errors due to a perceivable change in data quality.

In this case, we should run cycled experiments in low resolution to get
the new estimation for CrIS NPP (We had previously verified that the
difference in error estimation from high-resolution and low resolution
is not significant. So we can use the low-resolution estimation in 
high-resolution run).

Plan-B
For the best-case scenario where the data quality from side-1 is good and
has similar characteristics as those from side-2, we can do the following:
(1) turn off water vapor channels (set iuse flag to -2; do not use the data at all).
(2) use the existing obs error matrix and get rid of the rows/columns that 
       correspond to H2O channels
(3) run a single-cycle experiment with conditions from (1) and (2)
      together and validate
(4) we will run a cycled experiment if time and resources allow.  

To do list:

• [x] Prepare global_satinfo.txt file for the following changes: turn on CrIS NPP channels in LW band; set MW band channels to -2 (do not use; no bias coefficient update); and set SW band channels to -1 (monitoring; bias coefficient update for passive channels)
• [x] Perform single-cycle test for a cycle before the CrIS NPP problem (Plan-B)
• [x] Update correlated observation error matrix (see explanation why this is necessary from @KristenBathmann-NOAA)
• [x] Perform a drill parallel experiment based on v16x_sept_ctl to test the updated satinfo and Rcov_crisnpp files (Sanity Check: gsistat plots}
• [x] Run high-resolution single-cycle run with the real-time LW+SW data, updated satinfo, and the revised correlated obs error file (extended tasks)
• [x] Create the RadMon plots for this parallel run and also replot the operational RadMon plots with longer time range to include the data from May 2021 (@HaixiaLiu-NOAA)
• [x] Prepare abias and abis_pc (see Notes below) --- We will initialize CrIS NPP bias coefficients from zero
• [x] Turn on the assimilation of IASI-C (using IASI-B correlated observation error for IASI-C)
• [x] Run short high-resolution parallel experiment (1) update satinfo use flags for both CrIS NPP and IASI-C with the following updated configuration (2) use updated Rcov_crisnpp (3) use IASI-B Rcov for IASI-C (4) zero bias/bias_pc for CrIS NPP -[ ] Estimate Rcov for IASI-C using the high-resolution parallel run

Notes for abias and abias_pc (bias and pc files)

Background:

• We set the use flag for CrIS NPP in satinfo file to -1. This set all CrIS NPP channels to passive channels for monitoring and so the bias coefficients and pre-conditioning values are being estimated and updated unless the use flag is set to -2 (do not use) or no data flow at all.

• The NESDIS turned off the data stream entirely from 20210521 18Z, and then turned on the data (MW+SW) again from 20210602 12Z. The following describes the evolution of the bias and pc for CrIS NPP after NESDIS turned on MW+SW (1) LW channels: the bias coefficients stays the same as the 20210521 12Z (right before NESDIS turning off the data stream the pc values are set to default (10000); a large value since the data count is zero (2) MW channels: the bias coefficients were updated (since iuse_rad is set to -1) since the MW channels were available the pc values were estimated since the data count is not zero and greater than a pre-set threshold (3) SW channels: the bias coefficients were updated (since iuse_rad is set to -1) since the SW channels were available the pc values were estimated since the data count is not zero and greater than a pre-set threshold MW and SW channels are pass channels (not used in the assimilation at all). However, the bias estimation for these passive channels was not valid since LW channels were missing for QC (cloud detection). This won't impact the assimilation result.

What to do with the bias and pc values for CrIS NPP? --- we are going for Method#3

Method #1 Give NCO an updated bias and pc files with changes in CrIS NPP only Use the bias and pc values for CrIS NPP channels from the cycle (20200521 12z) right before the CrIS NPP side-2 issue

Method #2 Give NCO an updated bias and pc files with changes in CrIS NPP only (1) For LW channels: copy the bias coefficients and pre-conditioning values from the cycle (20200521 12Z) right before the CrIS NPP side-2 issue (2) For MW and SW channels set both bias coefficients and pc to zero (3) Remove CrIS NPP diagnostic file from the radstat (for the first cycle)

Method #3 Give NCO an updated bias and pc files with changes in CrIS NPP only (1) Start bias and pc estimation from zero for CrIS NPP, if this is allowed in the operational system (2) Remove CrIS NPP diagnostic file from the radstat (for the first cycle)

We better do a drill run for a few cycles before we hand these changes to NCO.

[RussTreadon-NOAA]

Prior to gfs.v15.1.1, NCO had the GFS in three main directories: gfs, gdas, and global_shared.

Tags for the DA components of GFS implementations v13.0.1 to v14.1.7 are in two directories:

• https://svnwcoss.ncep.noaa.gov/global_shared/tags/
• https://svnwcoss.ncep.noaa.gov/gdas/tags/

The global_shared directory contains GSI source code, scripts, jobs, and fix files. The gdas directory contains source code, scripts, jobs, and fix files for the ensemble part of the GFS DA package.

[KateFriedman-NOAA]

So at yesterday's meeting @RussTreadon-NOAA , @HaixiaLiu-NOAA, @CatherineThomas-NOAA and myself decided to relax the requirement for a definitive list of changes but to retain the gfsv16_historical/ directory to help with running retrospectives for the v16.x implementation only.

Do you guys still want to adjust the global_*txt file if-blocks in config.anal global-workflow or leave them as is and have users adjust their own config.anal files based on the GSI 0readme contents for now? Looking to wrap up the workflow side for this implementation. Thanks!

[ADCollard]

So at yesterday's meeting @RussTreadon-NOAA , @HaixiaLiu-NOAA, @CatherineThomas-NOAA and myself decided to relax the requirement for a definitive list of changes but to retain the gfsv16_historical/ directory to help with running retrospectives for the v16.x implementation only.

Do you guys still want to adjust the global_*txt file if-blocks in config.anal global-workflow or leave them as is and have users adjust their own config.anal files based on the GSI 0readme contents for now? Looking to wrap up the workflow side for this implementation. Thanks!

@KateFriedman-NOAA , I'll get you the workflow changes this morning.

[RussTreadon-NOAA]

Tag release/gfsda.v16.1.5 at c96150b as gfsda.v16.1.5.