YAML Configuration | OMI

[danholdaway]

Agreement with GSI using UFO

• [x] AURA

  Agreement with GSI using Swell

• [x] AURA

Current issues:

• None

[karpob]

@danholdaway I'm thinking the issues with ozone instruments might need to be reworked? We don't bias correct ozone obs. Also, I'm not exactly sure exactly what the first checkbox is. Running the ufo standalone with GSI GeoVaLs, and making sure it compares with the GSI output? If so, we can check some of those off, because I did that a year ago+ for OMI, and OMPS-NM. I don't think we did that exactly for MLS, but we did run it within FV3-JEDI.

[karpob]

@danholdaway @kwargan I think I know why the IODA converter doesn't get the geovals from the ncdiags. I have to take a look, but my guess is the IODA converter is expecting the ozone variable name to have ppmv in it or something. What is output from the GSI is labelled as "mass_concentration_of_ozone_in_air". So, I think the geovals are there, but they're not the name the converter is expecting, nor are the units correct (well, given how the GSI does things, the units will never be "correct", but I digress).

[danholdaway]

OK great. I think it could be quite useful if the ufo tests could be run for those instruments without too much effort. But if it seems like a lot of reworking of GSI is needed maybe it isn't worth it?

[karpob]

I think it might be possible to make a mod to the IODA converter to take on the mass concentration and convert it. It might take some tweaking depending upon what the GSI/UFO thinks that conversion factor is. I think it was consistent by x0044, but I'm not 100% sure.

[karpob]

I'll have a look at it, because I'm debugging some converter stuff anyway.

[karpob]

@danholdaway I can play with this also, but if you want to try it, I made a mod which I think works. https://github.com/JCSDA-internal/ioda-converters/tree/feature/ozone_geoval_geos

I tested it on one file and it outputs a geoval file with sane geovals for ozone in ppmv.

@kwargan Not sure if the value I got from the current gsi is consistent with what x044 was at the time. o3_conv = 603447.6

[danholdaway]

Thanks @karpob. The changes look sensible to me. If you have a file with sensible looking GeoVaLs is it worth putting together a UFO test (with or without QC) to check it works? If so then you could make a PR with these changes to ioda-converters.

[kwargan]

Getting an error in EvaDriver when running swell 1.0.5 with OMI. No plots get generated. The error log says: [A lot of things followed by] File "/discover/nobackup/drholdaw/opt/core/miniconda/3.9/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 167, in _nc4_require_group raise OSError("group not found: %s" % key, e) OSError: [Errno group not found: GsiEffectiveQC] 'GsiEffectiveQC' 2022-06-08T17:41:00Z CRITICAL - failed/ERR

[kwargan]

@kwargan Not sure if the value I got from the current gsi is consistent with what x044 was at the time. o3_conv = 603447.6

It's consistent with the current(ish) version of the GSI and I don't think that number has changed since the Cretaceous

[danholdaway]

OK. Let me regenerate the Observations files using the latest ioda-converters and see if that variable makes its way in.

[karpob]

No, Ricardo changed it recently. If you look at the commented out number, that's the one since the Cretaceous.

[kwargan]

Oh I didn't know that. I looked in the tag we're working with for R21C. Not sure what tag that is

[karpob]

! real(r_kind),parameter:: constoz = 604229.0_r_kind ! Where did this come from?

[karpob]

I don't think that's necessarily my problem with testing the UFO. I'm double checking to make sure the data necessary to run is in there. Something seems off.

[kwargan]

I'm finding this value (604229.0_r_kind) in a pretty old version of the GSI. No idea what tag x0044 used, though.

[danholdaway]

Ok I've updated all the ozone files and reverted them to what we had before. They are under the experiment called x0044_jjin_20220520. It seems that ioda-converters does not add GsiEffectiveQC to the converted files for ozone. I'm not sure why that would be the case. Perhaps it was only added for radiances and conventional. Or perhaps the ncdiags do not contain what ioda-converters needs in order to write GsiEffectiveQC. What you can do is comment out this line: https://github.com/GEOS-ESM/swell/blob/455187613cca50af5e361b681de9fba044bb29f8/src/swell/suites/hofx/eva.yaml#L15

[danholdaway]

I'll push some changes to remove the reading on GsiEffectiveQC by default so that the code works for ozone. I'll also turn an ozone instrument on by default so this kind of thing will be captured in testing going forward.

[kwargan]

Thanks, Dan. I shall try that now. Do I need to change anything to get the data from x0044_jjin_20220520 or is it the default in 1.0.5?

[danholdaway]

That's the default

[karpob]

@danholdaway OK, I'm definitely missing something here with just the default UFO (not looking at swell or anything). Looking at the default test for OMI straight out of the box (I don't change it to my files, just what is in the testinput directory), and I change the tolerance from 1e-4 to 1e-8, all test still pass. Does the UFO only test TLM/ADJ now? e.g. I run ctest ctest -R test_ufo_linopr_atminterplay_omi_aura -VV And no matter what I do to the tolerance, it passes. This seems broken, or I just don't understand what the tests do anymore.

[karpob]

@danholdaway sorry, running the wrong test. I want opr not linopr.

[kwargan]

@danholdaway OMI works for me now (although it needs more testing) but OMPS-NM fails in JediConfig: ABORT JediConfig: In find_instrument_from_string the string swell-experiment.ompsnm_npp.2020-12-14T21:00:00Z.nc4does not contain one of the valid instruments: where OMPS-NM appears to be there but listed as {'ioda name': 'ompstc8_npp', 'full name': 'OMPSTC-8 (NPP)'}

We can revisit this next week. It's getting late

[danholdaway]

Could there be a problem with naming convention in this file: https://github.com/GEOS-ESM/swell/blob/develop/src/swell/configuration/observation_ioda_names.yaml

[kwargan]

Instead of

• ioda name: ompstc8_npp full name: OMPSTC-8 (NPP) I would do
• ioda name: ompsnm_npp full name: OMPS Nadir Mapper (NPP) Same convention as for OMPS-NP (the preceding entry)

[danholdaway]

OK. I'll make the change as I turn add the eva fix

[karpob]

@danholdaway @kwargan So I'm pretty sure it won't be possible to run the stand alone UFO by running a modified ioda converter to take in the ozone profiles. There's something odd about how the geoval air_pressure_levels are filled in the ncdiags. There are NaNs, zeros, etc that just break things. I tried to filter them out, but I think the fill values sometimes get read in as "something." I might be able to salvage a subset, but it's really kind of a train wreck. I suggest we drop it.

[danholdaway]

OK. Thanks for looking carefully into it @karpob. If you think it best to drop the effort that sounds good. You can build swell using the branch in this PR https://github.com/GEOS-ESM/swell/pull/91 and it should work again for ozone instruments. The PR turns MLS and OMI on by default for all runs so they shouldn't break again. I didn't turn on OMPS as the Yaml isn't there yet. But once you/@kwargan have it ready feel free to turn on by default.

[kwargan]

I'll push some changes to remove the reading on GsiEffectiveQC by default so that the code works for ozone. I'll also turn an ozone instrument on by default so this kind of thing will be captured in testing going forward.

Dumb question: how do I update my swell to include these changes?

[danholdaway]

You can grab 1.0.6 of swell now. It should have everything you need to run with ozone

[karpob]

I added back the check-box for UFO, since we can run the tests now.

Initially, the test failed and the differences between UFO and GSI were several orders of magnitude higher than they should be (still small). The problem stems from NOAA using an old version of the GSI with an inaccurate value for constoz used to produce GeoVaLs for ozone concentration, along with a wrong integration constant (constants in the yaml) to simulate total ozone column in Dobson Units (DU). Based on first principles, Dobson units are calculated as

Following the above, the correct line for coefficients is

coefficients: [0.0078976797]

Before fix with UFO testing:

After fix:

Similar can be seen in Swell, just with more scatter.

Before fix:

After fix: (insert image here)

Will update with Swell scatter plot, and pull request shortly.