Preparation for turning on CrIS NPP radiances and include IASI Metop-C in operational GFS system

[emilyhcliu]

Timeline of Issus and Actions related to CrIS NPP

• 20210521 around 11:30 EDT: CrIS NPP began to show side-2 LW band problem
• 20210521 18Z: NESDIS turned off the CrIS NPP data stream (no CrIS NPP data at all)
• 20210528 12Z: NCO implemented satinfo change (set all CrIS NPP channels to -1)
• 20210602 12Z: NESDIS began to distribute side-2 MW+SW (no CrIS NPP LW data at all)
• 20210712: NESDIS began the process of switching CrIS NPP from side-2 to side-1
• 20210714: NESDIS completed the switch of CrIS NPP to side-1 and began the evaluation process
• 20210721: NESDIS completed the evaluation and held a review meeting (slides).
  The side-2 LW+SW data is in good shape. Data quality is comparable to those from side-1 before. After the side switch occurs, SNPP CrIS data will only be available from CLASS and GRAVITE. These data will be suspended from NDE/PDA until cleared by the calibration/validation science team.
• NOW: we need to prepare for using CrIS NPP again.

Official messages from NESDIS regarding the status CrIS NPP data

Preparation for CrIS NPP (switch back to side-1 for LW + SW)

Plan-A
The plan to cope with the scenario that we have to re-estimate the
correlated observation errors due to a perceivable change in data quality.

In this case, we should run cycled experiments in low resolution to get
the new estimation for CrIS NPP (We had previously verified that the
difference in error estimation from high-resolution and low resolution
is not significant. So we can use the low-resolution estimation in 
high-resolution run).

Plan-B
For the best-case scenario where the data quality from side-1 is good and
has similar characteristics as those from side-2, we can do the following:
(1) turn off water vapor channels (set iuse flag to -2; do not use the data at all).
(2) use the existing obs error matrix and get rid of the rows/columns that 
       correspond to H2O channels
(3) run a single-cycle experiment with conditions from (1) and (2)
      together and validate
(4) we will run a cycled experiment if time and resources allow.  

To do list:

• [x] Prepare global_satinfo.txt file for the following changes: turn on CrIS NPP channels in LW band; set MW band channels to -2 (do not use; no bias coefficient update); and set SW band channels to -1 (monitoring; bias coefficient update for passive channels)
• [x] Perform single-cycle test for a cycle before the CrIS NPP problem (Plan-B)
• [x] Update correlated observation error matrix (see explanation why this is necessary from @KristenBathmann-NOAA)
• [x] Perform a drill parallel experiment based on v16x_sept_ctl to test the updated satinfo and Rcov_crisnpp files (Sanity Check: gsistat plots}
• [x] Run high-resolution single-cycle run with the real-time LW+SW data, updated satinfo, and the revised correlated obs error file (extended tasks)
• [x] Create the RadMon plots for this parallel run and also replot the operational RadMon plots with longer time range to include the data from May 2021 (@HaixiaLiu-NOAA)
• [x] Prepare abias and abis_pc (see Notes below) --- We will initialize CrIS NPP bias coefficients from zero
• [x] Turn on the assimilation of IASI-C (using IASI-B correlated observation error for IASI-C)
• [x] Run short high-resolution parallel experiment (1) update satinfo use flags for both CrIS NPP and IASI-C with the following updated configuration (2) use updated Rcov_crisnpp (3) use IASI-B Rcov for IASI-C (4) zero bias/bias_pc for CrIS NPP -[ ] Estimate Rcov for IASI-C using the high-resolution parallel run

Notes for abias and abias_pc (bias and pc files)

Background:

• We set the use flag for CrIS NPP in satinfo file to -1. This set all CrIS NPP channels to passive channels for monitoring and so the bias coefficients and pre-conditioning values are being estimated and updated unless the use flag is set to -2 (do not use) or no data flow at all.

• The NESDIS turned off the data stream entirely from 20210521 18Z, and then turned on the data (MW+SW) again from 20210602 12Z. The following describes the evolution of the bias and pc for CrIS NPP after NESDIS turned on MW+SW (1) LW channels: the bias coefficients stays the same as the 20210521 12Z (right before NESDIS turning off the data stream the pc values are set to default (10000); a large value since the data count is zero (2) MW channels: the bias coefficients were updated (since iuse_rad is set to -1) since the MW channels were available the pc values were estimated since the data count is not zero and greater than a pre-set threshold (3) SW channels: the bias coefficients were updated (since iuse_rad is set to -1) since the SW channels were available the pc values were estimated since the data count is not zero and greater than a pre-set threshold MW and SW channels are pass channels (not used in the assimilation at all). However, the bias estimation for these passive channels was not valid since LW channels were missing for QC (cloud detection). This won't impact the assimilation result.

What to do with the bias and pc values for CrIS NPP? --- we are going for Method#3

Method #1 Give NCO an updated bias and pc files with changes in CrIS NPP only Use the bias and pc values for CrIS NPP channels from the cycle (20200521 12z) right before the CrIS NPP side-2 issue

Method #2 Give NCO an updated bias and pc files with changes in CrIS NPP only (1) For LW channels: copy the bias coefficients and pre-conditioning values from the cycle (20200521 12Z) right before the CrIS NPP side-2 issue (2) For MW and SW channels set both bias coefficients and pc to zero (3) Remove CrIS NPP diagnostic file from the radstat (for the first cycle)

Method #3 Give NCO an updated bias and pc files with changes in CrIS NPP only (1) Start bias and pc estimation from zero for CrIS NPP, if this is allowed in the operational system (2) Remove CrIS NPP diagnostic file from the radstat (for the first cycle)

We better do a drill run for a few cycles before we hand these changes to NCO.

[HaixiaLiu-NOAA]

@RussTreadon-NOAA Thank you for helping figure out the issue and provide the solution. I will follow your suggestion to make the changes.

[RussTreadon-NOAA]

@HaixiaLiu-NOAA : We should wait for Shelley's guidance. It is not clear to me if the change in operations is the ARFC mentioned in the SDM log or a temporary patch.

[ShelleyMelchior-NOAA]

The path forward: RFC 8521 updated to include changes to obsproc_prep_RB-5.5.0 obsproc_prep.v5.4.0a will run in operations until scheduled RFC 8521 implementation, 20210818 @1135Z

If real-time global paras need to test marine T2B changes, then use updated obsproc_prep_RB-5.5.0. If real-time global paras need to use operational obsproc_prep, then use obsproc_prep.v5.4.0a.

[HaixiaLiu-NOAA]

Following @RussTreadon-NOAA 's guidance, the following section is implemented in the config.base since 2021081306 cycle prep step.

export HOMEobsproc_prep="$BASE_GIT/obsproc/gfsv16b/obsproc_prep_RB-5.4.0a" export HOMEobsproc_network="$BASE_GIT/obsproc/gfsv16b/obsproc_global_RB-3.4.0"

After RFC 8521 implemented (1135Z on 8/18) use HOMEobsproc_prep and

HOMEobsproc_network below

if [[ "$CDATE" -ge "2021081806" ]]; then export HOMEobsproc_prep="$BASE_GIT/obsproc/obsproc_prep_RB-5.5.0" export HOMEobsproc_network="$BASE_GIT/obsproc/obsproc_global_RB-3.4.2" fi

After the above change was made, the 2021081306 gdasprep step ran to complete so the cron for v16iasicris2 has been re-enabled on Mars already.

[RussTreadon-NOAA]

v16iasicris2 update

A check of v16iasicris2 gdasprep.log confirm a successful switched to the RFC 8521 ObsProc upgrades effective 2021081806 gdas.

Sufficient v16iascris2 cycles have been completed to stand up a monitoring and verification page. METplus is now the official EMC monitoring and verification package. However, METplus does not yet incorporate fit2obs. While METplus can generate a scorecard, the procedure to do so involves more steps (and accounts) than vsdb based scorecards. Given this, the v16iasicris2 monitoring and verification driver executes vsdb, fit2obs, and METplus plotting jobs. A cron has been set up under emc.glopara to run daily at 1245Z. v16iasicris2 results are plotted along with the operational gfs. Details follow

The script executed by cron is /gpfs/dell2/emc/modeling/noscrub/emc.glopara/para_fv3gfs/v16iasicris2/scripts/generate_plots.sh.

VSDB, fit2obs, and a scorecard are on the on the web at https://www.emc.ncep.noaa.gov/gmb/emc.glopara/vsdb/v16iasicris2/

METplus plots, including links to fit2obs, are on the web at https://www.emc.ncep.noaa.gov/gmb/emc.glopara/metplus/v16iasicris2/

Please note that v16iasicris2 verification is not available for all forecast hours given the short duration, so far, of this parallel.

[HaixiaLiu-NOAA]

@RussTreadon-NOAA Thank you for making these plots.

I also made the gsistat plots shown below.

[HaixiaLiu-NOAA]

The parallel experiment v16iasicris2 has a starting date 2021081212 and has run in real-time since 8/22. Its VSDB stats have not enough days yet, but the HGT AC for day1-3 in the tropics show worse scores. I will watch the stats in the following days.

[HaixiaLiu-NOAA]

The RadMon plots were plots from this high-reso para. Below is the link

https://www.emc.ncep.noaa.gov/gmb/wx20hl/radmon.v16iasicris2/

[HaixiaLiu-NOAA]

The # of obs passing QC for IASI_Metop-C

The # of obs passing QC for CrIS_NPP

[ADCollard]

@KristenBathmann-NOAA Do you have the MetOp-B IASI plots for reference as I am not clear which channels are of concern. The radmon plots look almost identical for the two instruments.

[KristenBathmann]

I was double inflating the window channels. I will delete my previous post. Here are the covariance results for IASI-C from Haixia's high res run (after reconditioning and variance inflation): Here are the differences between Haixia's high res run and my low res run:

Everything looks as I would expect it to. In the plots of the correlation matrices differences, the channels that are shaded pink actually have very small cross-correlations, but they are non-negligible. We see the biggest differences in channels that have surface sensitivity.

[HaixiaLiu-NOAA]

Kristen provided updated Rcov_iasicsea and Rcov_iasicland computed from the high-reso parallel experiment v16iasicris2. The para v16iasicris2 will use these two new Rcov files starting from 2021090212 cycle (the global_anavinfo.l127.txt has been updated to use new Rcov). I will examine the gdasanal.log file to make sure the analysis at 2021090212 uses proper Rcov.

[HaixiaLiu-NOAA]

The gdasanal step is running now on WCOSS. I checked the log file, and believe it is using the Rcov correctly.

Below is what I greped from the log file (per Kristen's suggestion):

Rcov(Evals) for Instrument: iasi_metop-c:sea cond= 6.7955322990E+01 Rcov(stdev) for instrument: iasi_metop-c:sea recond Rcov(Evals) for Instrument: iasi_metop-b:sea cond= 6.8648778493E+01 Rcov(stdev) for instrument: iasi_metop-b:sea recond Rcov(Evals) for Instrument: iasi_metop-c:land cond= 1.3134032549E+02 Rcov(stdev) for instrument: iasi_metop-c:land recond Rcov(Evals) for Instrument: iasi_metop-b:land cond= 1.5388534575E+02 Rcov(stdev) for instrument: iasi_metop-b:land recond Rcov(Evals) for Instrument: cris-fsr_n20:sea cond= 3.6394468822E+01 Rcov(stdev) for instrument: cris-fsr_n20:sea recond Rcov(Evals) for Instrument: cris-fsr_npp:sea cond= 2.4364200152E+01 Rcov(stdev) for instrument: cris-fsr_npp:sea recond

Kristen has kindly confirmed it is correct.

[HaixiaLiu-NOAA]

Kate created a release branch for this work off of the operations branch: release/gfsv16.1.4

[HaixiaLiu-NOAA]

The parallel has been handed over to EIB on 9/2/2021. Russ has kindly helped with the transition to EIB, figured one important issue with the workflow used by this parallel at gldas step. The v1.12.0 gldas should be updated to v1.13.0 in order to pick up right precipitation forcing file if running in catch-up mode (fall behind real-time).

[HaixiaLiu-NOAA]

Mike has created a new GSI branch release/gfsda.v16.1.4 for me to merge the changes in the para v16iasicris2 and prepare for the gfsv16.1.4 implementation

[HaixiaLiu-NOAA]

the branch release/gfsda.v16.1.4 has been updated @ ea47071 with new fix files

[CatherineThomas-NOAA]

Because of the emergency fix for Metop-A IASI, the tentative v16.1.4 implementation is being renumbered to v16.1.5. Release branch release/gfsda.v16.1.4 has been renamed release/gfsda.v16.1.5.

[HaixiaLiu-NOAA]

merged release/gfsda.v16.1.4 (turn off Metop-A IASI, modify read_nsstbufr for new ships data) into release/gfsda.v16.1.5. Update global_satinfo.txt to turn OFF CrIS_NPP LW channels for now since NESDIS is going to perform some activity on the instrument.

[HaixiaLiu-NOAA]

commit # is a4b17f6

[HaixiaLiu-NOAA]

@RussTreadon-NOAA Tagged release/gfsda.v16.1.5 at a4b17f6 as gfsda.v16.1.5

Informed EIB with a release notes sent for them to tag a gfsv16.1.5 (see global-workflow issue #439

[HaixiaLiu-NOAA]

EIB (Lin Gan) reported that the parallel v16iasicris2 failed at gdas fcst and efcs for 2021092612 cycle with segmentation fault. Checking the gdasanal.log shows the following abnormal numbers for excess moisture at the 2nd outer loop. background 1.1088734627843344E+05 excess moisture 2.8187805775106544E+09 surface pressure 8.5934129105349984E+03 temperature 3.3566234523096457E+04 wind 1.7544417993326945E+05 moisture 4.0724491979939771E+03 sst 1.2770620102454295E+03 ozone 8.8503332717725498E+03 gps bending angle 1.3466609115552888E+05 radiance 1.0815256294795012E+06 tcp (tropic cyclone) 7.9107996054366922E+01

J Global 2.8203395393574109E+09 End Jo table inner/outer loop 0 2 Initial cost function = 2.820339539357410908E+09

@RussTreadon-NOAA recognized this issue is similar to what he has encountered with the operational job having poor minimization performance at 2021090818 (minimization resets in the second outer loop). Russ explained as follows.

"The TLNMC produced a large decrease (order 40 to 80K) in the temperature at a few points need the top of the model. This decreased qsat by several orders of magnitude. The q>qsat (excess moisture) part of limq strongly forced the minimization. Other J terms forced just as hard in the other direction. John placed a lower bound of qmin on the qsat computed in genqsat. This allowed the 2021090818 case to have a much better behaved minimization."

Russ copied the v16iasicris2 GSI source codes and modified the genqsat.f90 with John's change and ran a test for the 2021092612 cycle. This time the fcst and efcs steps finished without problem. The analysis log file does not show huge excess moisture either for this test run. Russ then ran several more tests for this case. Below is what Russ found.

"Running the master global_gsi.x on Venus with verbose=.true. yielded the following interesting output when running setuprad to generate obs-anl stats (o-g 03).

CRTM_Atmosphere_IsValid(INFORMATION) : Negative layer temperature found CRTM_K_Matrix(FAILURE) : Input data check failed for profile #1 crtm_interface*call_crtm: ERROR during crtm_k_matrix call 3

It was not generated when using the v16iasicris2 global_gsi.x. The v16iasicris2 global_gsi.x is taken from gfs.v16.1.2 operations. This version is many commits behind the master. The master genqsat.f90 contains the qmin bound. When this bound is removed from the master, the crtm negative temperature error is printed at the end of the analysis when we are computing obs-anl stats. The final mean analysis and temperature increments printed to stdout clearly show a problem

Status Var Mean Min Max analysis TSEN 6.110027604983E+01 -7.197790316214E+01 3.162091384706E+02 increment tsen 6.225808814011E-03 -8.620771751332E+01 1.793439705164E+02

-7.2 K is not a physical temperature. The min/max temperature increments are -86.2 to 17.9 K. -86.2 K is a large temperature increment.

The v16iasicris2 does not have generated a negative temperature but it generated unphysically low temperatures. The final table or mean analysis and increment values has

Status Var Mean Min Max analysis TSEN 6.110095248790E+01 7.291699160979E+00 3.161370802150E+02 increment tsen 3.043870615047E-03 -4.259829624494E+01 9.371398875238E+01

We should not see a minimum temperature of 7.3 K with min/max increments of -42.6 to 93.7 K.

When we add the qmin bound in genqsat the final analysis stats are

Status Var Mean Min Max analysis TSEN 6.110170242654E+01 1.189764009672E+02 3.161366563238E+02 increment tsen 4.558658543681E-04 -2.162717946370E+01 1.126794659782E+01

119.0 K is still quite cold but it's not as extreme as 7 K"

Will need to discuss with related DA persons on how to fix this

[HaixiaLiu-NOAA]

Discussed the minimization issue from the v16iasicris2 parallel with @RussTreadon-NOAA, @emilyhcliu @ADCollard @KristenBathmann-NOAA. The decision was made to put a lower bound of qsat in the genqsat.f90 in the gfsda.v16.1.5 to avoid extreme cold temperature being generated.

[RussTreadon-NOAA]

Recreate tag gfsda.v16.1.5 as copy of release/gfsda.v16.1.5 at 7c5af91.

[HaixiaLiu-NOAA]

Thank you, @RussTreadon-NOAA for updating the tag.

@KateFriedman-NOAA Would you please update the global-workflow with the updated GSI tag: gfsda.v16.1.5?

[HaixiaLiu-NOAA]

I will update the release notes accordingly and send it to you, @KateFriedman-NOAA

[RussTreadon-NOAA]

global-workflow branch release/gfs.v16.1.5 has already been updated to DA tag gfsda.v16.1.5. @KateFriedman-NOAA did this at eabced1.

[KateFriedman-NOAA]

I will update the release notes accordingly and send it to you, @KateFriedman-NOAA

@HaixiaLiu-NOAA Okie dokie, ready to ingest them when ready, thanks!

[RussTreadon-NOAA]

Impact of TLNMC modes for 2021092618 gdas

Run 2021092618 gdas case with various number of vertical modes, NVMODES_KEEP, retained by the TLNMC. Operations and the v16iasicris2 run with NVMODES_KEEP=8 Tabulated below are the final (min,max) sensible temperature analysis and increments (K) along with the global_gsi.x wall time (seconds). All jobs ran the global_gsi.x on Mars Phase 3 nodes (dev) with 420 tasks, ptile=7, and 4 threads per task.

NVMODES_KEEP	min TSEN_anl	max TSEN_anl	min TSEN_inc	max TSEN_inc	wall time
8	-7.197208336965E+01	3.162091326562E+02	-8.620760370897E+01	1.793433683642E+02	4784.132894
16	-2.453231766415E+05	1.807245904767E+05	-2.489749882328E+05	1.563747923358E+05	3528.656965
32	1.545324084268E+02	3.161397330673E+02	-1.095196881517E+01	9.224151897082E+00	5930.787827
64	1.655445993616E+02	3.161260995374E+02	-8.269896903773E+00	7.442405664575E+00	6509.331250

The NVMODES_KEEP=16 minimization aborted in the second outer loop at iteration 34 with the message

 PCGSOI: WARNING **** Stopping inner iteration ***
 Penalty increase or constant   2   34  0.826323728307584750E+15  0.826323728307584625E+15

This explains the low wall time for this configuration.

The gdas and enkf forecasts from the NVMODES_KEEP=8 run seg faulted shortly after execution began. Forecasts were not run from the NVMODES_KEEP=32 or NVMODES_KEEP=64 cases. Since these configurations retained positive analysis temperatures it is expected that the forecasts, if run, would have run to completion.

None of the above gsi runs placed a lower bound of qmin=1.0e-07 on the qsat computed in genqsat.f90. Adding the line
qsat(i,j,k) = max(qmin,qsat(i,j,k)) to genqsat.f90 and rerunning the 2021092618 gdas case with NVMODES_KEEP=8 yields the following sensible temperature results:

NVMODES_KEEP	min TSEN_anl	max TSEN_anl	min TSEN_inc	max TSEN_inc
8	1.190163096796E+02	3.162124911269E+02	-2.720773128018E+01	1.418466351114E+01

The numbers above come from the v16iasicris2 after updating the GSI source code. The 2021092618 gdas fcst ran past the point of the previous failures. Similar success is expected with the enkf forecasts.

[HaixiaLiu-NOAA]

Updated Release Notes for gfsv16.1.5

[HaixiaLiu-NOAA]

1.190163096796E

@RussTreadon-NOAA Should the above number be 1.190163096796E+02? Setting a lower bound of qmin=1.0e-07 on the qsat in genqsat.f90 results in relatively reasonable tsen_anl while not requiring a lot more wall time. EIB resumed the v16iasicris2 parallel and all the jobs have completed without issue at the 2021092612 cycle. The gdasanal.log at this cycle shows reasonable analysis and increment values.

[HaixiaLiu-NOAA]

@RussTreadon-NOAA Thank you for running those tests which provide helpful wall time guidance if the number of retained vertical modes increases.

[RussTreadon-NOAA]

1.190163096796E

@RussTreadon-NOAA Should the above number be 1.190163096796E+02? Setting a lower bound of qmin=1.0e-07 on the qsat in genqsat.f90 results in relatively reasonable tsen_anl while not requiring a lot more wall time. EIB resumed the v16iasicris2 parallel and all the jobs have completed without issue at the 2021092612 cycle. The gdasanal.log at this cycle shows reasonable analysis and increment values.

Yes, @HaixiaLiu-NOAA , you are correct. Thank you for reading the comment and pointing out my error. The table has been corrected.

[HaixiaLiu-NOAA]

The GFS.v16.1.5 will be implemented on October 26 at 1430Z (RFC 8823).

[HaixiaLiu-NOAA]

RFC 8823 - WCOSS: Upgrade to GFS v16.1.5 for GSI changes ... @ Tue Oct 26, 2021 10:30 - 15:30 (EDT), Start to implement this RFC. 1930Z - This RFC is completed.

[HaixiaLiu-NOAA]

@RussTreadon-NOAA @ADCollard My understanding is we should add an additional historical fix file to the gfsda.v16.1.5 release branch fix/gfsv16_historical/global_satinfo.2021102612 following the GFS v16.1.5 implementation. Is that right?

[RussTreadon-NOAA]

Yes, this is the pattern we have been following for the past few implementations. As you say, we cp fix/global_satinfo.txt fix/gfsv16_historical/global_satinfo.2021102612.

To which fix branch do we commit? The most complete approach is to commit to fix branch DA_GFSv16.1.5_global_only and then merge into rev2. If we commit to DA_GFSv16.1.5_global_only we should recreate tag gfsda.v16.1.5. One could also just commit to rev2 since this is the fix master.

Your question raises a larger question. What is the purpose of files in fix/gfsv16_historical? Is the intention that the dated files in this directory reflect when each file was implemented in operations? This is what the above accomplishes. fix/gfsv16_historical is a historical record of operational global fix files changes.

Another possibility is that we want fix/gfsv16_historical to reflect when to start using various observation types in retrospective parallels. Developers, not NCO, use the contents of fix/gfsv16_historical. Currently global-workflow configuration files config.anal and config.prep reference fix/gfsv16_historical via $RUN_ENVIR = "emc" blocks. These EMC only sections, in turn, contain logical tests which compare $CDATE with date ranges.
This allows us to toggle between various fix files as a retrospective parallel cycles forward.

There is a time lag between when we test updated fix files and when the changes are implemented. The $CDATE appended to files in fix/gfsv16_historical could reflect the date at which we determine it is safe, even desirable, to use the given fix file in retrospective parallels.

Whatever the meaning of the $CDATE we append to files in fix/gfsv16_historical, we need to update parm/config/config.anal and/or parm/config/config.prep accordingly. A check of the global-workflow develop parm/config/config.anal and parm/config/config.prep shows that this has not yet been done for GFS v16.1.5. Should we make the same changes to global-workflow tag EMC-v16.1.5?

My personal preference is to do away with fix/fv3_historical and fix/gfsv16_historical. Instead we have a source controlled usage database. This database tracks when we can/should use a given data type along with the operational evolution of usage for the given data type. The workflow queries the usage database for the given $CDATE and builds info and error files on the fly.

An additional keyword could be set in a configuration file to specify how we build the info and error files. dev or para mode mode means we use the data at the earliest date at which it is safe to use the data. opr or prod mode means we use the data when it was used in operations. If NCO prefers to use static fix files instead of building files on the fly, we create a snapshot of the fix files to be implemented. We currently do this with our DA_GFSvX.Y.Z_global_only branches in the fix repo.

Of course, the usage database approach has its pitfalls. How do we define the safe to use dates? These dates can (will) change as we develop new techniques to process data. Data usage varies from one DA system to another. Regional DA may (will) have different dates (both _safe_touse and implementation) than global DA. Similar comments apply for the CDAS, WDAS, etc. Do we have one usage database which covers all DA or multiple databases - one for each DA system?

We are moving away from GSI based DA. What will be the JEDI approach to addressing this topic?

[KateFriedman-NOAA]

Whatever the meaning of the $CDATE we append to files in fix/gfsv16_historical, we need to update parm/config/config.anal and/or parm/config/config.prep accordingly. A check of the global-workflow develop parm/config/config.anal and parm/config/config.prep shows that this has not yet been done for GFS v16.1.5. Should we make the same changes to global-workflow tag EMC-v16.1.5?

I'm looking to wrap up the v16.1.5 global-workflow work this week and merge release/gfs.v16.1.5 into the operations branch. I'm happy to ingest config changes for the fix file if-blocks before submitting the PR, let me know what you'd like to do. Thanks!

[CatherineThomas-NOAA]

Going back a bit further, the corresponding config changes were also not included for v16.1.3 (IASI-A) and v16.1.4 (DO-3). DO-2 is already in. Let me know if I can help.

[ADCollard]

@RussTreadon-NOAA, for what it's worth, the original intent of these files was to replicate usage in operations so that operations can be used as a control.

I think your suggestion of a database is the way forward. We need to start the conversation within the JEDI community on how to achieve this.

[ADCollard]

Unless someone else is volunteering to do it, I am happy to commit the changes.

So if I understand what Russ is saying we should first commit to DA_GFSv16.1.5_global_only and retag before merging with the master?

[RussTreadon-NOAA]

@ADCollard , thanks for the reminder. This agrees with the $CDATE suffix for most, if not all, the files in fix/gfsv16_historical. We need to ensure the workflow properly toggles between various historical fix files. At times developers may want to bypass this logic. This can be done in their personal checkout of the workflow.

The steps you outline sound right. @HaixiaLiu-NOAA noted that we need to add fix/gfsv16_historical/global_satinfo.2021102612. This is a copy of fix/global_satinfo. This gets committed to the fix branch you mention, DA_GFSv16.1.5_global_only.

From here we could update the fix submodule in release/gfsda.v16.1.5 and create a new gfsda.v16.1.5 tag, This isn't necessary but it's nice for completeness. We could skip this step and directly merge DA_GFSv16.1.5_global_only into rev2 and update the master fix submodule.

PR #237 has a DA Review Committee deadline of 11/1/2021. Pending a green light form the committee, Mike could begin working on PR #237 tomorrow. Fully closing PR #237 includes updating the fix and libsrc modules. We could work with Mike and combine the above v16.1.5 fix file addition with v16.x fix file changes.

[MichaelLueken]

When working with the fix submodule, I can begin by merging DA_GFSv16.1.5_global_only into DA_GFSv16.x_global_only, then merge DA_GFSv16.x_global_only into rev2 (wrapping up the fix submodule update).

Please comment on this issue once the file has been added to DA_GFSv16.1.5_global_only, then I will move forward with the fix submodule updates.

[ADCollard]

Looking in fix/gfsv16_historical of DA_GFSv16.1.5_global_only, I'm seeing global_satinfo.txt.202109220 which appears to be the v16.1.3 implementation and global_convinfo.txt.2021091612 which appears to be the v16.1.4 implementation.

The date associated with the latter appears to be when data became available rather than when it went operational (2021092712). Do we agree that this date should be changed?

[CatherineThomas-NOAA]

I'm okay with changing the date for DO-3. I went by the date first available, as I also did for DO-2 (global_convinfo.txt.2021031712).

Should all files be changed for their operational date? For instance, the earliest satinfo file here, global_satinfo.txt.2019021900, includes instruments not implemented until March 2021 with v16 (AHI Himawari-8, ABI G16, ATMS Ch. 15, etc). It seems like this would get messy very fast. My memory for this particular v16 directory was not for matching operations, but for running the pre-implementation retrospective parallels.

[RussTreadon-NOAA]

We have competing ideas for what historical fix files represent and how they are used. One approach is that they faithfully record the cycle at which the given fix file was implemented. This approach has an operational focus. The historical files are a record of operational data usage. Another approach is that info files represent the cycle at which it is safe to use data in the given file. This approach focuses on testing and development. We maximize the use of data in retrospective parallels. One directory of files can't support both options.

On Mon, Nov 1, 2021 at 1:57 PM CatherineThomas-NOAA < @.***> wrote:

I'm okay with changing the date for DO-3. I went by the date first available, as I also did for DO-2 (global_convinfo.txt.2021031712).

Should all files be changed for their operational date? For instance, the earliest satinfo file here, global_satinfo.txt.2019021900, includes instruments not implemented until March 2021 with v16 (AHI Himawari-8, ABI G16, ATMS Ch. 15, etc). It seems like this would get messy very fast. My memory for this particular v16 directory was not for matching operations, but for running the pre-implementation retrospective parallels.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/GSI/issues/186#issuecomment-956453426, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGNN634HBMV5ABBIC4I2JNLUJ3IJFANCNFSM5AYQB5WQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[ADCollard]

So things are definitely a little messy. And I am beginning to think that I do not have the proper perspective on the v16 implementation.

Firstly, there are two directories: fv3_historical and gfsv16_historical. fv3_historical has not been updated since Jan 2020 (for an ozone change) with satinfo and convinfo changes last done in 2018.

I was assuming gfsv16_historical replaced the functionality of fv3_historical, but I guess now that the intent was to use the former for the v16 parallels.

So should we update fv3_historical to reflect the more recent operational upgrades?

[ADCollard]

Can I suggest we have a quick meeting to discuss this?

[ADCollard]

So at yesterday's meeting @RussTreadon-NOAA , @HaixiaLiu-NOAA, @CatherineThomas-NOAA and myself decided to relax the requirement for a definitive list of changes but to retain the gfsv16_historical/ directory to help with running retrospectives for the v16.x implementation only.

To that end, I have updated the DA_GFSv16.1_global_only fix branch to add new satinfo file to gfsv16_historical/ (reflecting the turning on of MetOp-C IASI on 2021102612) and have added a 0readme file to this directory that documents the changes between each of the historical conv/oz/satinfo files.

I have also removed the fv3_historical/ directory as that was becoming more and more out of date.

@MichaelLueken-NOAA, I believe this is ready to move forward with the fix submodule updates.

[RussTreadon-NOAA]

NCO creates implementation tags in their svn repository. Tags for GFS implementations are in

https://svnwcoss.ncep.noaa.gov/gfs/tags/

As of 11/2/2021 this directory has GFS tags from gfs.v12.0.2 to gfs.v16.1.5. GFS versions 12 to 14 do not include the GFS DA components. GFS DA, including DA fix files, came under the gfs directory structure starting with gfs.v15.1.1.

The gfs.v16.1.5 tag contains DA fix files in both the workflow location, fix/fix_gsi, and in the sorc/gsi.fd fix directory.