Input observation error

[danholdaway]

There were some recent commits to gsi_ncdiag.py that have broken our ability to convert ncdiags on the GMAO side, I'm guessing also on the EMC side but I haven't tested there. I think this may be a bug. Specifically at line 1130 it is now requesting 'input_observation_error' from radiance files, which is not a variable we have, and I don't think is a variable that the EMC version of GSI outputs either. https://github.com/JCSDA/ioda-converters/blob/ff76cecc8263ddfb040b20623285fe28ae08a868/src/gsi-ncdiag/gsi_ncdiag.py#L1130

Is it supposed to be inverse_observation_error?

[danholdaway]

@markjolah please assign anyone else that may be able to help, I'm not sure who works on this the most at the moment.

[CoryMartin-NOAA]

@danholdaway @emilyhcliu looks like it was added here: https://github.com/JCSDA/ioda-converters/pull/254 I remember Emily saying it was because the value we were using before was not correct. Is it simply a matter of merging in changes to GSI from feature/files_for_jedi?

[danholdaway]

@emilyhcliu @CoryMartin-NOAA Is this something that has always been written to the GSI binary diags? Since at GMAO we are looking at running a converter from the binary diags to ncdiags and then the to ioda converter. We are going off the operational system so merging new things into gsi isn't really an option.

[CoryMartin-NOAA]

@danholdaway Emily would know more than I. I have never used the binary diags. Perhaps the converter will have to be modified? Or else we can add logic to the GSI ioda-converter to fall back if that variable doesn't exist? It may only be an issue for all-sky radiances.

[danholdaway]

I was looking at some of the GSI diag files, not sure how up to date they are, and it seems some files have input_observation_error, but not all of them.

[srherbener]

Could someone please add an estimate to this issue. Thanks!

[emilyhcliu]

@danholdaway The "Input_Observation_Error" field saves the original observation error assignment. The dimension is number of channels. It contains the error assignment save in the following nc_diag field: do i=1, nchanl n=ich(i) : call nc_diag_chaninfo("error_variance", varch(n)) enddo

This is Ok for instruments assimilated under clear-sky condition. The same set of original observation error is applied to all observation locations.

However, for all-sky assimilation such as AMSU-A, the original observation error assignment is a function of retrieved cloud liquid water that varies with locations. So, the observation error assignment needs to be the function of location and channel.

[emilyhcliu]

@danholdaway Are you using the information saved in the "Input_Observation_Error" in the assimilation? My guess is not, since it contains original observation error and what you need is the final observation error in the assimilation.

Are you using the GSI feature branch: feature/files_for_jedi?

[danholdaway]

I see, so in order to have Input_Observation_Error we need to be using feature/files_for_jedi in the GSI. Since we aren't using that branch, the diags from from the operational system, is there a workaround? Can we put 1/Inverse_Observation_Error instead? Are you saying that would only impact All Sky?

[emilyhcliu]

@danholdaway I see, so you have your own GSI branch for data conversion.
Yes, putting 1/inverse_observation_error works since the dimension is location and channel.
In ioda-converter, the "Input Observation Error" field is now a function of location and channel for all instruments. IASI for example, it will be the same set of obs error values for all locations.

[danholdaway]

@will-mccarty brining your attention to this thread regarding the email I sent you. Looks like we can do the 1/inverse hack for now.

[markjolah]

@danholdaway I suspect this is resolved by now?

[danholdaway]

Yes, it is resolved.