Open danholdaway opened 9 months ago
The follow figures are the same as those figures with black dots in the above PDF file:
JJ: these are somewhat consistent w/ what I see when I make corresponding figs from my runs - which I showed last Wed. What worries me most right now are not these difference, but the diffs I see in observation errors and how, in some case, PS (for sound) and most of the radiance, R seems completely off no matter what I do.
for example ... I showed last week R gsi vs jedi for PS:
and here is what I get for AMSUA-N15
I believe there is some error somewhere when it comes to the observation error assignment.
Agree. Definitely there is something wrong with AMSUA.
These are all the 2D GeoVaLs for AMSUA-N19, 2021-12-12T00:00:00. JEDI uses a C90 background and GSI comes from x0048 ncdiags.
NOTE PLots removed. See below for updated versions.
Agree. Definitely there is something wrong with AMSUA.
It's not just AMSU-A ... I can make similar figs for everything ...
These are all the 2D GeoVaLs for AMSUA-N19, 2021-12-12T00:00:00. JEDI uses a C90 background and GSI comes from x0048 ncdiags.
Dan: the vegetation_type_index seems to me to be the most suspect one; followed by wind-speed; and FOV ... but vegtype-index looks as though on code considers something to be 1 and the other 0 or something like that, no?
Thank you both. The atmospheric mode resolution is C360 in x0048. I think a part of these differences can be attributed to the difference between C90 vs C360. Don't you agree?
@rtodling I cannot see your figures here.
Jianjun,
Try accessing the plots from Dan's original email from 9:28pm yesterday....those work for me while the same list in Ricardo's email does not.
Ron
From: Jianjun Jin @.> Sent: Thursday, December 14, 2023 10:21 AM To: GEOS-ESM/swell @.> Cc: Gelaro, Ronald (GSFC-6101) @.>; Assign @.> Subject: [EXTERNAL] [BULK] Re: [GEOS-ESM/swell] Evaluate the fv3-jedi GeoVaLs against fv3-jedi GeoVaLs (Issue #241)
CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.
@rtodlinghttps://github.com/rtodling I cannot see your figures here.
— Reply to this email directly, view it on GitHubhttps://github.com/GEOS-ESM/swell/issues/241#issuecomment-1856046149, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AXWQP7JMYAKOEJFD7GRHT7DYJMKOXAVCNFSM6AAAAAA5DLUO22VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJWGA2DMMJUHE. You are receiving this because you were assigned.Message ID: @.***>
Dan,
Thanks for extracting these. Obviously some of these could be at the root of our problem. From my perspective, here are the fields I think show significant disagreement, even after consideration of the difference between the JEDI and GSI resolutions:
And one that might be questionable but could be ok:
Thanks, Ron
From: rtodling @.> Sent: Thursday, December 14, 2023 9:41 AM To: GEOS-ESM/swell @.> Cc: Gelaro, Ronald (GSFC-6101) @.>; Assign @.> Subject: [EXTERNAL] [BULK] Re: [GEOS-ESM/swell] Evaluate the fv3-jedi GeoVaLs against fv3-jedi GeoVaLs (Issue #241)
CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.
These are all the 2D GeoVaLs for AMSUA-N19, 2021-12-12T00:00:00. JEDI uses a C90 background and GSI comes from x0048 ncdiags.
average_surface_temperature_within_field_of_view
ice_area_fraction
land_area_fraction
leaf_area_index
soil_temperature
soil_type
surface_geopotential_height
surface_snow_area_fraction
surface_snow_thickness
surface_temperature_where_ice
surface_temperature_where_land
surface_temperature_where_sea
surface_temperature_where_snow
surface_wind_from_direction
surface_wind_speed
vegetation_area_fraction
vegetation_type_index
volume_fraction_of_condensed_water_in_soil
water_area_fraction
Dan: the vegetation_type_index seems to me to be the most suspect one; followed by wind-speed; and FOV ... but vegtype-index looks as though on code considers something to be 1 and the other 0 or something like that, no?
— Reply to this email directly, view it on GitHubhttps://github.com/GEOS-ESM/swell/issues/241#issuecomment-1855975298, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AXWQP7O4VW5POHGUHTQKIITYJMF2JAVCNFSM6AAAAAA5DLUO22VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJVHE3TKMRZHA. You are receiving this because you were assigned.Message ID: @.***>
It would be good to run without the resolution reduction to C90. I'll try that next
That would be great, thanks.
Ron
From: Dan Holdaway @.> Sent: Thursday, December 14, 2023 12:56 PM To: GEOS-ESM/swell @.> Cc: Gelaro, Ronald (GSFC-6101) @.>; Assign @.> Subject: [EXTERNAL] [BULK] Re: [GEOS-ESM/swell] Evaluate the fv3-jedi GeoVaLs against fv3-jedi GeoVaLs (Issue #241)
CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.
It would be good to run without the resolution reduction to C90. I'll try that next
— Reply to this email directly, view it on GitHubhttps://github.com/GEOS-ESM/swell/issues/241#issuecomment-1856328031, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AXWQP7IUXGGCXIIVUZQFELDYJM4TFAVCNFSM6AAAAAA5DLUO22VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJWGMZDQMBTGE. You are receiving this because you were assigned.Message ID: @.***>
hofx rerun with C360 background. Adding level 50 plots for 3D GeoVaLs. Modified the difference plot of surface wind from direction to use angular difference.
I know that we'll start looking at the GEOVALS comps again. I'm going to say this: when I look at the plots above and think about the differences I believe the diffs are all - or mostly - related to a resolution choice.
Remember that the geovals that come from GSI have all been derived from x0048; which has backgrounds at 576x361. The JEDI results are all derived from c90 backgrounds in the r2d2-swell test. Most of the fields above are surface fields; they will be extremely influence by resolution.
When we start looking at these plots - and trying to generate them again - we must use c180 backgrounds; those will be the ones consistent w/ what GSI does.
Ricardo, I hope you're right and agree we'll want to do the comparison at comparable resolution to take that off the table. But I admit I'm not as certain how to reassure myself that the diffs above are all or mostly due to resolution. We can talk about this on Wednesday too.
Ricardo, I hope you're right and agree we'll want to do the comparison at comparable resolution to take that off the table. But I admit I'm not as certain how to reassure myself that the diffs above are all or mostly due to resolution. We can talk about this on Wednesday too.
I am setting things up to do this test ... this can't just easily be set from swell.
@asewnath : can you pls help us all look at the GeoVals once again?
I placed two sets of geovals here:
GSI-geovals:
/gpfsm/dnb33/rtodling/SwellExperiments/GeoValComp/swell-convert_ncdiags/run/20211212T000000Z/geos_atmosphere
and JEDI-geovals using c180 backgrounds here: /gpfsm/dnb33/rtodling/SwellExperiments/GeoValComp/swell-hofx/run/20211212T000000Z/geos_atmosphere/c180
Thank you so much.
The idea would be to product comp plots as those posted by Dan some time ago - this is basically a redo of what Dan did, but he used c360 backgrounds which I don't think are consistent w/ the resolution used in GSI. Also, given the fixes in the fix surface fields we have implemented since then, we should see at least three or so differences improve. We should also be able to get a sense for how things look in general.
I would also ask that we don't necessarily post results here - at least not until we browse at results in more mundane ways; then what we should do is extract the troubled fields and post them here for the record so we can start working on at least understanding and/or correcting each one.
Many thanks.
All: I am quite convinced that the issues we have with the geovals related to the surface fields and associated with the code below:
I am finding hard to see one-to-one match between this code and what our version of GSI has. There is some connection, but the wrapping is so large that it requires very close examination; this can benefit from many of us looking at it carefully.
I am just realizing that GSI does not put out a geoval for CO2. This means that whatever we are seeing in the plots comparing CO2 geoval from JEDI to GSI is fishy - it's probably cooked up in the converters! I bet there is logic in the ncdiag converters that fill CO2 w/ some value when it doesn't find it in the ncdiag files.
This geoval should be added to GSI to allow for proper comparison.
if we did not put out a groval for co2 properly, how could we get the match between GSI and JEDI when we did hofx testing?
I think I got this figured out.
I am plugging below similar figs to what Dan plugged above, but now with fixes I see fit to address the discrepancies.
Basically, this relates to the fact that the vegetation model used by GMAO-GSI had not been implemented in the crtm-interface (fv3-jedi). There a couple of other things too that related to differences that seem to be small bugs in fv3-jedi: one of which is the handling of the surface wind-speed, which JEDI/fv3-jedi is scaling by f10m whereas it should not be doing so.
The 3d-fields are plotted at level 50 just as in Dan's original figures. With these, the plots above show largely noticeable differences only in:
average_surface_temperature_within_field_of_view and mole_fraction_of_carbon_dioxide_in_air
Fortunately, there seems to be an explanation for such. Neither of these are being put out in the ncdiag files that GMAO's GSI generates (x0048); therefore, it is my understanding that the ncdiag converters fill in something for the "missing" fields; which makes the difference plots meaningless.
The one other fields I have some concern with is
leaf_area_index
I think it might be worthwhile looking at its calculation in JEDI a little more closely and comparing it with GSI's.
Unfortunately, there is one critical difference that the figures above don't show. This only reveals it self if we look at low-index levels. The issue is with air_pressure. For example, creating the figures for level index 30 shows the following undesirable difference in air_pressure:
Interestingly, air_pressure_levels does not show relevant differences.
Looking a bit more closely, this seems to be associated with the fact that GSI and JEDI are using different definitions for mid-layer pressure. Unlike air_pressure_levels for which Vader implements two options, Vader only implements a single option for air_pressure (mid-level pressures): i.e., Phillips interpolated; the GMAO GSI uses the more mundane mid-level calculation: mean of pressure_above plus pressure_below mid level. This will be made into a Vader issue.
The fixes used to produce the figures above are now in:
I know where the difference in pressure comes from. GMAO-GSI used mean-layer for mid-lev pressures, whereas JEDI is using the Phillips-based formulation of mid-levels (this is available in GSI) it is not the option we use.
I know how to address this ... currently under testing.
The issue is more complicated that simply the use of a different calculation for mid-level pressures. The issue is associated with how GSI adds levels to the pressure coordinate passed on to CRTM.
This is handled in the GSI's routine add_rtm_levels and the interface and level pressures that come out of it are not those associated directly with the underlying model pressures.
At this point I am not completely sure how best to handle this and in any case, this will need to involve others in the discussion who can justify what GSI is doing in this regard.
The simplest why to see this is to look at level 1 (top) and we'll see that GSI's (from the perspective of geovals/radiance) has the mid-level top near the top of CRTM: TOA_PRESSURE=0.005 (hPa). JEDI on the other hand is coming up w/ a lower value since it calculating the mid level wrt to the top of GEOS, 0.01 hPa.
There are just so many more problems with the GeoVals ... just looking at GeoVals from GMI ... there are things in the GSI operator that don't seem to be in UFO ... I'll be more specific tomorrow.
Wow, Ricardo, you've done a tremendous amount of work sussing all this out. The plots you posted are so much better than our previous go-around. Understood that there many more problems. But now at least you seem to pinpointed what most of them are. Thanks for spending your Saturday on this. Let's talk more tomorrow.
Here is another puzzle. This geoval comes out of AURA OMI ... this is a level index 30 (upper troposphere lower strat).
and lower down in the troposphere
No sure yet how this translates into ozone increment differences.
I am going to correct a comment I made above about CO2. I said that I don't see CO2 being written out by GSI in the ncdiag files and I was wondering if the converters make something up. That's not quite correct.
GSI does write all all CRTM absorbers (including CO2) down to the ncdiag files ... I missed that part when I looked for it in setuprad ... my mistake. Now that's no so good because it means that something is not adding up when it comes to the CRTM absorbers wrt to CO2 ... I looked at O3 from the IASI operator and that looks ok.
Now here is another questionable one ... and I think I know where that is handled ... Tropopause pressure ... lev=30 confirmed to be the same in every level (as it should be).
The Model2GeoVals entry:
tropopause pressure method: gsi
should address this ...
I think the gsi tropopause method is the default ... using the toggle above gives me the same odd result ... looking more closely now.
Just for reference and to double check that there is indeed a bug in the conversion of the GSI routine to JEDI (tropprs), I tried the alternative option to use Greg Thompson's implementation of the tpause pressure ... not supposed to be identical to GSI's but ...
much closer ... the question now is: where is the bug in the ported routine from GSI? ... looking ...
@rtodling CO2 (mole_fraction_of_carbon_dioxide_in_air) is a constant, 407 ppmv, in the output of FV3-JEDI. It is likely because of this line: https://github.com/JCSDA-internal/fv3-jedi/blob/develop/src/fv3jedi/VariableChange/Model2GeoVaLs/fv3jedi_vc_model2geovals_mod.f90#L545
@rtodling We get a constant in CO2 because "xm%has_field('co2')= F" in https://github.com/JCSDA-internal/fv3-jedi/blob/develop/src/fv3jedi/VariableChange/Model2GeoVaLs/fv3jedi_vc_model2geovals_mod.f90#L546 We have CO2 in GEOS, don't we? Not sure why it cannot be found.
Here is a PR to update the tropopause estimate by the GSI method. https://github.com/JCSDA-internal/fv3-jedi/pull/1207
Thank you, Jinajun!
Ron
From: Jianjun Jin @.> Sent: Thursday, May 23, 2024 11:18 AM To: GEOS-ESM/swell @.> Cc: Gelaro, Ronald (GSFC-6101) @.>; Assign @.> Subject: [EXTERNAL] [BULK] Re: [GEOS-ESM/swell] Evaluate the fv3-jedi GeoVaLs against GSI GeoVaLs (Issue #241)
CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.
Here is a PR to update the tropopause estimate by the GSI method. JCSDA-internal/fv3-jedi#1207https://github.com/JCSDA-internal/fv3-jedi/pull/1207
— Reply to this email directly, view it on GitHubhttps://github.com/GEOS-ESM/swell/issues/241#issuecomment-2127403075, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AXWQP7JLBBOXQOOBGBYVM4TZDYCC7AVCNFSM6AAAAAA5DLUO22VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRXGQYDGMBXGU. You are receiving this because you were assigned.Message ID: @.***>
Here is a PR to update the tropopause estimate by the GSI method. JCSDA-internal/fv3-jedi#1207
As discussed in https://github.com/JCSDA-internal/fv3-jedi/pull/1208 - TROPPRS is not an issue in JEDI - the problem was found to be in GMAO's interface to GSI - the problem is now fixed:
Here is another puzzle. This geoval comes out of AURA OMI ... this is a level index 30 (upper troposphere lower strat).
There is no inconsistency in ozone from OMI/Aura. The differences we observed are caused by the order of vertical levels in the geoval files between GSI and JEDI, as shown in the following plot for the corresponding pressure at index 30:
in other words, the order of vertical levels is top-down in the geoval file from JEDI, while it is bottom-up in GSI. So index 30 in JEDI corresponds to index 44 in GSI for pressure(totaling 73 levels). As a result, the differences in both pressure and ozone between GSI and JEDI now become much smaller:
Hi @gmao-wgu , I've actually known that reverse in the levels - I had meant to tell you that this was the number one thing to look into. Sorry that I didn't, but thank you so much for figuring out.
At this point - since we all understand what's going on - I would say, let's not take any action on this. We could certainly patch GSI and reverse the levels in all of the 1d (or 2d) geovals; but I'd say, let's not worry about that right. We got other fish to fry.
But again - thank you so much for the diagnose.
@rtodling Differences are overall small between JEDI and GSI GNSSRO geoval fields. It looks like that some differences at lower levels (level # 50, which is ~500hPa here) can be attributed to resolution differences. However, the JEDI results are indeed made with a resolution of C180 in Swell-hofx test. Therefore, resolution is not a factor for the differences.
@asewnath I made some updates in your Python plot code because the levels are bottom-up in conventional GSI goeval files and they are always top-down in fv3-jedi outputs. See /gpfsm/dnb34/jjin3/SwellExperiments/validate_geoval/org_compare_jedi_geovals_plots.py . Thanks again for your plotting program.
Dan gets the credit for the plotting program :)
Here are comparisons between results made by fv3-jedi in "swell_hofx" and results made in "swell-ufo_testing" using Geoval made by GEOS. swell_geoval_sonde.pdf These figures are color coded: Orange: results made in "swell-ufo_testing" using Geoval made by GSI. Cyan: results made by fv3-jedi in "swell_hofx". These are results which @rtodling's produces, but at a different date. Black: compare results made by fv3-jedi and GEOS Geoval on which we should focus. Red: comparisons of GSI results int the two Swell tests. These are used to verify that Swell tests results are re-arranged correctly in order to compare them at the same locations.