search

ESMValGroup / ESMValTool

ESMValTool: A community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP
https://www.esmvaltool.org
Apache License 2.0
217 stars 126 forks source link

Review of Autoassess metrics: Stratosphere #1193

Closed valeriupredoi closed 5 years ago

valeriupredoi commented 5 years ago

Review of the Autoassess area metrics/recipes in ESMValTool (associated code changes PR #1192 )

The Autoassess recipes are clearly labelled in the ESMValTool/esmvaltool/recipes repository (we are not reviewing the _rms_ recipes since they were ported by @valeriupredoi from ESMValTool v1 to v2): https://github.com/ESMValGroup/ESMValTool/tree/version2_development/esmvaltool/recipes

Agreed parameters:

Run location: Jasmin: /home/users/valeriu/esmvaltool_cmip6_autoassess

fx files: CMIP5 sftlf files were used instead of CMIP6 since they are not yet available for CMIP6.

ESMValTool version: 2.0.0a1; /home/users/valeriu/anaconda3Feb19/envs/esmv/bin/esmvaltool

ESMValTool Results: https://github.com/NCAS-CMS/NCAS-Useful-Documentation/tree/master/autoassess_review_results

Met Office result pages:

valeriupredoi commented 5 years ago

Stratosphere results comparison so far

Stratosphere area generally stands the comparison well, in terms of numbers, for UKESM1 historical experiment the metrics are:

Polar night jet: southern hem (July), A: 112.05  V: 84.86 shit
100 hPa 10Sto10N temp (annual mean),A: 15.46  V: 15.43 ok
Easterly jet: southern hem (January),A: 75.85  V: 24.85 shit
QBO amplitude at 30 hPa (eastward),A: 17.36  V: 4.65 shit Vtrop: 17.32 OK
50 hPa temperature: 60N-90N (MAM),A: 40.92  V: 40.95 ok
50 hPa temperature: 90S-60S (SON),A: 23.63  V: 23.9 ok
100 hPa equatorial temp (annual cycle strength),A: 1.666  V: 1.668 ok
Easterly jet: northern hem (July),A: 55.74  V: 29.87 shit
50 hPa temperature: 90S-60S (JJA),A: 11.30  V: 11.10 ok
Summary,A: 11.9998  V: 9.316 shit  Vtrop: 11.18 OK
100 hPa 10Sto10N temp (annual cycle strength),A: 1.619  V: 1.623 ok
Polar night jet: northern hem (January),A: 44.91  V: 40.33 shit-ok
50 hPa temperature: 60N-90N (DJF),A: 26.85  V: 26.75 ok
QBO amplitude at 30 hPa (westward),A: 27.39  V: 1.62 shiiit Vtrop: 27.38 OK
QBO period at 30 hPa,A: 41.0  V: 19.75 shit Vtrop: 41.5 OK
100 hPa equatorial temp (annual mean),A: 15.30  V: 15.29 ok
70 hPa 10Sto10N wv (annual mean),A: 5.75  V: 5.74 ok

(legend: A: Alistir's run at the MO, V: V's run at Jasmin, Vtrop: V's run with QBO computed on Tropics belt); there are still some problems with pnj_metrics

valeriupredoi commented 5 years ago

landsurface_surfrad has now been updated with correct land fraction mask (my previous results I was selecting oceans instead of land mass :man_facepalming: ) https://github.com/NCAS-CMS/NCAS-Useful-Documentation/tree/master/autoassess_review_results/landsurface_surfrad_correctedMask/plots

valeriupredoi commented 5 years ago

Note the documentation for Autoassess recipes has now been started to be ported: https://github.com/ESMValGroup/ESMValTool/pull/1198

alistairsellar commented 5 years ago

Reviewing stratosphere outputs...

Reviewing plots:

  1. qbo_30hpa.png - OK
  2. teq_100hpa.png - OK
  3. t100_vs_q70.png - not OK: It looks like you are plotting every month, perhaps every gridpoint too, so missing the spatio-temporal meaning. There should be just one point for each model (see https://collab.metoffice.gov.uk/twiki/bin/viewfile/Static/AlistairSellar/AutoAssess/u-bc179_v_u-aw310_2005_010/stratosphere/thumb_t100_vs_q70.png).
  4. zonal mean latitude-pressure plots - almost OK but not quite: they look great for the pressure range plotted, but they should extend to 0.1 hPa (=10 Pa). Can you try getting zonal mean U & T from MIP table AerMonZ instead of Amon? These should be on plev39, which I expect extend higher than plev19. Also, can you convert the y axis to hPa? Meteorologists aren't used to looking at Pa.

Reviewing metrics. As outlined so delicately above, all OK apart from:

Polar night jet: southern hem (July), A: 112.05  V: 84.86
Easterly jet: southern hem (January),A: 75.85  V: 24.85
Easterly jet: northern hem (July),A: 55.74  V: 29.87
Polar night jet: northern hem (January),A: 44.91  V: 40.33

These could potentially be solved by the levels issue too - what pressure range is the code asking for?

valeriupredoi commented 5 years ago

@alistairsellar man cheers ecer so much for looking at the results :beer:

Reviewing stratosphere outputs...

Reviewing plots:

qbo_30hpa.png - OK -- cool!
teq_100hpa.png - OK -- cool!
t100_vs_q70.png - not OK: It looks like you are plotting every month, perhaps every gridpoint too, so missing the spatio-temporal meaning. There should be just one point for each model (see https://collab.metoffice.gov.uk/twiki/bin/viewfile/Static/AlistairSellar/AutoAssess/u-bc179_v_u-aw310_2005_010/stratosphere/thumb_t100_vs_q70.png).

yes, I know that - note two things: the lime green patch differs since we don't have MERRA data so I used ERA-Interim instead; the blue and orange triangles are mean values over time indeed but not having the specific merra_tropical_area_avg.nc I used all the data available hence the end result tmean = t_mean(tmon) - t_merra is computed with all the lat-lon values from ERA-I, that's why you see more than one point; this is obviously wrong, so now I am introducing the same plev and tropics extraction and months collapse on the ERA-I data, will update the results as soon as the run finishes

zonal mean latitude-pressure plots - almost OK but not quite: they look great for the pressure range plotted, but they should extend to 0.1 hPa (=10 Pa). Can you try getting zonal mean U & T from MIP table AerMonZ instead of Amon? These should be on plev39, which I expect extend higher than plev19. Also, can you convert the y axis to hPa? Meteorologists aren't used to looking at Pa.

Unfortunately UKESM1 data on ESGF doesn't have ua data for AERmonZ :cry: OK Ill convert to hPa

Reviewing metrics. As outlined so delicately above, all OK apart from:

Polar night jet: southern hem (July), A: 112.05 V: 84.86 Easterly jet: southern hem (January),A: 75.85 V: 24.85 Easterly jet: northern hem (July),A: 55.74 V: 29.87 Polar night jet: northern hem (January),A: 44.91 V: 40.33

These could potentially be solved by the levels issue too - what pressure range is the code asking for?

Having a second (third) look at those right after I run the modified t100_q70 plot :beer:

valeriupredoi commented 5 years ago

t100_vs_q70.png has now been fixed and code changes pushed: https://github.com/NCAS-CMS/NCAS-Useful-Documentation/blob/master/autoassess_review_results/stratosphere_correctT100Q70/plots/aa_strato/autoassess_strato_test_1/UKESM1-0-LL-piCont_vs_UKESM1-0-LL-hist/stratosphere/t100_vs_q70.png

valeriupredoi commented 5 years ago

@alistairsellar for the life of me I can not understand why the pnj metrics differ for us since the only thing I changed in the main pnj metrics func and its auxiliary is the correct air pressure for iris cubes:

NOTE that by original version autoassess I mean the base I took from Paul Earnshaw on Jasmin: /home/users/pearnshaw/NewAutoAssess/aa_2018.04.01/autoassess/assessment_areas/stratosphere/strat_metrics_1.py that may differ from what you used. Can you pls have a look at the codes here and at your end? :beer:

alistairsellar commented 5 years ago

t100_vs_q70.png has now been fixed

Much better, but still not quite there...

  1. The top axis is inconsistent with the bottom one, such that model bias vs ERA-Interim is wrong: should be ~2.7 K - 3 K.
  2. Similar problem for right axis.
  3. The green rectangle is shifted relative to the two model points.

I think that all of these are side effects of not having MERRA data available, so I suggest that the best way to solve this is to make ERA-I the primary axes (bottom & left instead of top & right), and not have any secondary axes. Then define the green triangle using the data range it covers in the ERA-I coordinates: x-axis: 0.8 - 2.8 K y-axis: 0.1 - 0.85 ppmv

alistairsellar commented 5 years ago

Unfortunately UKESM1 data on ESGF doesn't have ua data for AERmonZ 😢 OK Ill convert to hPa

Great on the conversion to hPa.

Could you make the latitude - pressure plots for another model which does have ua and ta in AERmonZ? In terms of verifying the port, I'm happy that the UKESM plots you made are close enough to the originals up to 10hPa that the code is working correctly. But to be useful the plots really should go up to 0.1 hPa, so if you can switch the namelist to use AERmonZ and plot another model I'd be happy to sign that off.

alistairsellar commented 5 years ago

Can you pls have a look at the codes here and at your end?

Here are the same two functions from the version I used. They look the same as yours (though I haven't been through line-by-line)...

def pnj_strength(cube, winter=True):
    '''
    Calculate PNJ and ENJ strength as max/(-min) of zonal mean U wind
    for nh/sh in winter and sh/nh in summer repsectively.
    '''
    # Extract regions of interest
    notrop = iris.Constraint(pressure=lambda p: p < 80.0)
    nh_cons = iris.Constraint(latitude=lambda l: l > 0)
    sh_cons = iris.Constraint(latitude=lambda l: l < 0)
    nh_tmp = cube.extract(notrop & nh_cons)
    sh_tmp = cube.extract(notrop & sh_cons)
    # Calculate max/min depending on season
    coords = ['latitude', 'pressure']
    if winter:
        pnj_max = nh_tmp.collapsed(coords, iris.analysis.MAX)
        pnj_min = sh_tmp.collapsed(coords, iris.analysis.MIN) * (-1.0)
    else:
        pnj_max = sh_tmp.collapsed(coords, iris.analysis.MAX)
        pnj_min = nh_tmp.collapsed(coords, iris.analysis.MIN) * (-1.0)
    return (pnj_max, pnj_min)

def pnj_metrics(run, ucube, metrics):
    '''
    Routine to calculate PNJ strength metrics from zonal mean U
    Also produce diagnostic plots of zonal mean U
    '''

    # Extract U for January and average over years
    jancube = ucube.extract(iris.Constraint(month_number=1))
    jan_annm = jancube.collapsed('time', iris.analysis.MEAN)

    # Extract U for July and average over years
    julcube = ucube.extract(iris.Constraint(month_number=7))
    jul_annm = julcube.collapsed('time', iris.analysis.MEAN)

    # Calculate PNJ and ENJ strengths
    (jan_pnj, jan_enj) = pnj_strength(jan_annm, winter=True)
    (jul_pnj, jul_enj) = pnj_strength(jul_annm, winter=False)

    # Add to metrics dictionary
    metrics['Polar night jet: northern hem (January)'] = jan_pnj.data
    metrics['Polar night jet: southern hem (July)'] = jul_pnj.data
    metrics['Easterly jet: southern hem (January)'] = jan_enj.data
    metrics['Easterly jet: northern hem (July)'] = jul_enj.data

    # Plot U(Jan) and U(Jul)
    plot_uwind(jan_annm, 'January', '{}_u_jan.png'.format(run['runid']))
    plot_uwind(jul_annm, 'July', '{}_u_jul.png'.format(run['runid']))
valeriupredoi commented 5 years ago

Ali man:

The top axis is inconsistent with the bottom one, such that model bias vs ERA-Interim is wrong: should be ~2.7 K - 3 K.

The top and right axes don't mean anything in this plot since I am using ERA-I instead of MERRA (so the left and bottom axes are actual data but ERA-I); the ERA-I bits (top and right axes) are commented out of the code and those ticks are just stock Matplotlib ticks

Similar problem for right axis. The green rectangle is shifted relative to the two model points.

It actually is not, you are seeing it shifted since the plot has negative limits (Matplotlib artifact if you don't set xlim and ylim), the green rectangle occupies exactly the same coords as yours

So - the historical and piControl points are slightly shifted since the MERRA data was replaced with ERA-I only the plotting format has been kept as if I used MERRA data, that's why it appears different, but the values are consistent ie:

q bias wrt ERA: ~1.75 ppmv t bias wrt ERA-I: ~2.75 K

valeriupredoi commented 5 years ago

what I can do, to obv make things straightforward, is to completely take MERRA out from the labeling and replace it with ERA-I, and remove the top and right axes :smiling_imp:

alistairsellar commented 5 years ago

Yes, that's what I was suggesting. If you do that, you should also shift the green box. Its extent on the ERA-I axes should be: x-axis: 0.8 - 2.8 K y-axis: 0.1 - 0.85 ppmv

valeriupredoi commented 5 years ago

its extent is dictated by q_era:

    # Plot ideal area
    patch = Rectangle(
        (0.0, 0.0),
        2.0,
        0.2 * q_era,
        fc='lime',
        ec='None',
        zorder=0)
    ax1.add_patch(patch)

(converted from q_merra to q_era in the code) - is that not correct?

valeriupredoi commented 5 years ago

also just managed to figure out where the problem with the PNJ is:

    if winter:
        pnj_max = nh_tmp.collapsed(coords, iris.analysis.MAX)
        pnj_min = sh_tmp.collapsed(coords, iris.analysis.MIN) * (-1.0)
    else:
        pnj_max = sh_tmp.collapsed(coords, iris.analysis.MAX)
        pnj_min = nh_tmp.collapsed(coords, iris.analysis.MIN) * (-1.0)
    return (pnj_max, pnj_min)

since the data is extracted from air_pressure=lambda p: p < 8000.0 (in Pa, or pressure=lambda p: p < 80.0 in hPa for your native autoassess) the collapse on MAX should probably be the same but on MIN is much lower for your data that doesn't stop at 100 Pa as is the case with my CMOR data. So maybe we can call this OK as well since the code is identical but only the analyzed data varies?

alistairsellar commented 5 years ago

(converted from q_merra to q_era in the code) - is that not correct?

Well, it depends on the numbers used to do the conversion. But no matter whether you define the extent in "MERRA" coordinates and convert it, or define it directly in "ERA" coordinates, the extent of the box relative to the ERA axes should be: 0.8 - 2.8 K, 0.1 - 0.85 ppmv.

valeriupredoi commented 5 years ago

(converted from q_merra to q_era in the code) - is that not correct?

Well, it depends on the numbers used to do the conversion. But no matter whether you define the extent in "MERRA" coordinates and convert it, or define it directly in "ERA" coordinates, the extent of the box relative to the ERA axes should be: 0.8 - 2.8 K, 0.1 - 0.85 ppmv.

sorry man, I am not sure why that is that way, since the box corners are somewhat hardcoded in the plotting function - can you pls tell me what box bounds to use in:

    # Plot ideal area
    patch = Rectangle(
        (0.0, 0.0),
        x,
        y,
        fc='lime',
        ec='None',
        zorder=0)
    ax1.add_patch(patch)

is y now fixed? It was 0.2 * q before. See why an engineer shouldn't do the work of a scientist :grin:

valeriupredoi commented 5 years ago

is this correct? eraMerra

alistairsellar commented 5 years ago

The axes now look right. Try this for the box:

patch = Rectangle((0.8, 0.1), 2.8, 0.2*q_era+0.1, fc='lime', ec='None', zorder=0)

I also suggest adding the following comment to explain the odd-looking limits for the box:

# Arbitrary box of acceptability for Met Office model development, designed to target warm 
# tropopause temperature biases (e.g. Hardiman et al (2015) DOI: 10.1175/JCLI-D-15-0075.1. 
# Defined as T bias < 2K and q bias < 20% relative to MERRA. 
# MERRA is not used in this plot so ranges shifted by +0.8 K and +0.1 ppmv to account for
# differences between MERRA and ERA-Interim.
# todo: Make box symmetric about zero to be relevant to models with a cold bias?
valeriupredoi commented 5 years ago

dis gud? :beer: era2

valeriupredoi commented 5 years ago

also wrt:

Could you make the latitude - pressure plots for another model which does have ua and ta in AERmonZ? In terms of verifying the port, I'm happy that the UKESM plots you made are close enough to the originals up to 10hPa that the code is working correctly. But to be useful the plots really should go up to 0.1 hPa, so if you can switch the namelist to use AERmonZ and plot another model I'd be happy to sign that off.

I checked on Jasmin and could not find any model that has AERmonZ and has data for both ta and ua unfortunately pfftt

alistairsellar commented 5 years ago

dis gud? 🍺

Ah, the 2nd pair of args must be box size - I assumed they were end positions. As a result the box is too large. Specifying width / height instead should work then:

patch = Rectangle((0.8, 0.1), 2.0, 0.2*q_era, fc='lime', ec='None', zorder=0)

No change required to the explanatory comment.

valeriupredoi commented 5 years ago

how about dis? Don't mind it's pink, it's my screen that's acting up era2

alistairsellar commented 5 years ago

Hrrm, still slightly different from the original, but I can't understand why. Tell you what, let's drop the green box for now. It's a pretty arbitrary acceptance range, and very Met Office-specific. For initial inclusion in ESMValTool, perhaps it would be better to make it a generic plot comparing just the model values.

valeriupredoi commented 5 years ago

Ali man, we can always tweak things, we don't have to set this in stone; I mean, what's important is to decide now that this diagnostic/metric is good to be included in the v2.0 release (and the paper) and that's about it for now.

I also think that me doing the tweaking work is not the best idea since I have no scientific background - the software integration works well and the scientific tweaking is now trivial for someone who knows a bit of Python but more importantly know what the scientific result should look like

valeriupredoi commented 5 years ago

when I ported all these Autoassess metrics in to ESMValTool I specifically asked for scientific review and code changes from knowledgeable people - I initially asked Paul Earnshaw but his FTE in the project proved to be tiny so he dropped off the radar; and then I left things as they are because this is as much as my scientific knowledge can go, unfortunately

alistairsellar commented 5 years ago

Yes, I agree that things can be tweaked later. I think it is best for now to remove the green rectangle, and it can be added later when someone has the time to get it right. I'm not happy including it as it stands, because it is not correct. If you can remove the rectangle, I'll sign this one off.

valeriupredoi commented 5 years ago

removing it as we speak and Ill add the comments above and a pointer TODO to have it reinstated :beer:

what about the other question marks?

alistairsellar commented 5 years ago

what about the other question marks?

I think the last question marks are on the PNJ metrics and the latitude-height plots. Both of these could be solved by using plev39 from AERmonZ (you are right when you say above that the height limit is the reason for missing the max strength of the polar jets, as these are higher than 10hPa).

I just had a look, and UKESM does have AERmonZ data on JASMIN for some of its historical members. See e.g.

/badc/cmip6/data/CMIP6/CMIP/MOHC/UKESM1-0-LL/historical/r1i1p1f2/AERmonZ/ua/gnz/latest/ua_AERmonZ_UKESM1-0-LL_historical_r1i1p1f2_gnz_195001-201412.nc

But not for the piControl for some reason. So how about making the lat-pressure plots and PNJ metrics for just the UKESM historical, using AERmonZ? This will be enough to verify against the AutoAssess run I made. If it's easier to make a pairwise comparison, then you could use HadGEM3-GC31-LL r1i1p1f3 as the control instead.

valeriupredoi commented 5 years ago

running UKESM1-0-LL vs HadGEM3-GC31-LL right about now

valeriupredoi commented 5 years ago

for UKESM1-0-LL pnj's with ESMValTool (AERmonZ for ta and ua):

Polar night jet: northern hem (January),44.864559173583984
Polar night jet: southern hem (July),112.08815002441406
Easterly jet: southern hem (January),76.12091827392578
Easterly jet: northern hem (July),55.68126678466797

yours were

Polar night jet: northern hem (January),A: 44.91
Polar night jet: southern hem (July), A: 112.05
Easterly jet: southern hem (January),A: 75.85
Easterly jet: northern hem (July),A: 55.74

do we agree or do we agree? :grin:

valeriupredoi commented 5 years ago

and di ploti, which are totally identical to those in here: https://collab.metoffice.gov.uk/twiki/bin/viewfile/Static/AlistairSellar/AutoAssess/u-bc179_v_u-aw310_2005_010/stratosphere/stratosphere.html

PT-ukesm

alistairsellar commented 5 years ago

That'll do pig, that'll do.

alistairsellar commented 5 years ago

Great work!

How about u-bc179_t_djf.png and u-bc179_qbo.png? I figure these will look right with the AERmonZ data too...

valeriupredoi commented 5 years ago

yes - you run those :grin: I can not run those without heavy modifications since AERmonZ data comes only on time-air_pressure-latitude and all the other metrics apart from PNJ use area weights computed from iris.analysis that in turn, needs to have at least a scalar single-point for longitude. Hacking iris or creating my own eweights may induce a truckload of rrors, the metrics agree so there's no reason why anything else should not :beer:

alistairsellar commented 5 years ago

OK, I can see that's not-trivial if Iris needs a longitude coordinate. Can you add a TODO in the documentation page that the latitude-pressure T plots, and time-pressure QBO plots need to be updated to work with zonal mean inputs from AERmonZ?

Now that everything has been checked, I propose that you remove the special start and end arguments in the diagnostic namelist which you used to force the Dec-1 start & finish dates (mentioned in the "Time gating" section of the documentation page). This was useful to get a precise comparison against the AutoAssess plots & metrics, but is an unnecessary complication for future usage. Better just to select whole years in whatever is the standard ESMValTool way.

valeriupredoi commented 5 years ago

unfortunately that needs to be kept for now since we need to review the other metrics as well and they all use the same wrapper to make things easy (autoassess_area_base.py). We'll remove that once we have all in place, there are a bunch of other things I'd like to simplify as well. Note that I wrote the note about AERmonZ in the documentation :beer:

https://github.com/ESMValGroup/ESMValTool/blob/version2_development_autoassess_recipes_documentation/doc/sphinx/source/recipes/recipe_autoassess_stratosphere.rst

valeriupredoi commented 5 years ago

Q for you though: shouldn't you be an author or at least some reference of your name in the documentation? I took the MO names and references straight from the version from Paul but you are not there...

alistairsellar commented 5 years ago

unfortunately that needs to be kept for now since we need to review the other metrics

Fair enough. It would be good to state explicitly in the documentation that this is a temporary measure. Presumably you could have two wrappers / bases: one for the review stage and one for operational metrics which have been reviewed.

Note that I wrote the note about AERmonZ in the documentation

I only see one note, and it doesn't mention that there is a TODO to get the 2D plots working with zonal means from AERmonZ. The comment also implies that after this test that the PNJ metrics will be calculated using the plev17 data in Amon. Is that what you mean?

valeriupredoi commented 5 years ago

unfortunately that needs to be kept for now since we need to review the other metrics

Fair enough. It would be good to state explicitly in the documentation that this is a temporary measure. Presumably you could have two wrappers / bases: one for the review stage and one for operational metrics which have been reviewed.

clarified it now, cheers :beer: No, I really don't want to mess around with multiple versions of one code in the same arch version of the master code; it's easier if we just get all the metrics in, review and then remove the extra time selection

Note that I wrote the note about AERmonZ in the documentation

I only see one note, and it doesn't mention that there is a TODO to get the 2D plots working with zonal means from AERmonZ. The comment also implies that after this test that the PNJ metrics will be calculated using the plev17 data in Amon. Is that what you mean?

The question is: do we want to use AERmonZ everywhere in this diagnostic or not? Since if we use AERmonZ for one plot and one metric and something else for all the other plots and metrics then it's bad - we're mixing apples with oranges. I was under the assumption that the use of AERmonZ here was only for testing and then the user will use whatever mip they want, but one mip for the whole diagnostic. What say you?

alistairsellar commented 5 years ago

Thanks for the chat just now. Just to confirm that if you take a mean over latitude using iris.analysis.cartography.cosine_latitude_weights(cube) of the zonal mean data in AERmonZ, this will be equivalent to taking a lat-lon mean using iris.analysis.cartography.area_weights(cube) over the 3D data in Amon.

valeriupredoi commented 5 years ago

you sir are correct, running with AERmonZ for all metrics for ua and ta (and Amon for hus since hus is nowhere to be found for AERmonZ) I gets for UKESM1-0-LL historical:

Polar night jet: northern hem (January),44.864559173583984
Polar night jet: southern hem (July),112.08815002441406
Easterly jet: southern hem (January),76.12091827392578
Easterly jet: northern hem (July),55.68126678466797
QBO period at 30 hPa,41.5
QBO amplitude at 30 hPa (westward),27.398354323511228
QBO amplitude at 30 hPa (eastward),17.36300870021091
50 hPa temperature: 60N-90N (DJF),27.11177833755525
50 hPa temperature: 60N-90N (MAM),40.93849718531288
50 hPa temperature: 90S-60S (JJA),11.746028760414731
50 hPa temperature: 90S-60S (SON),23.881006628825844
100 hPa equatorial temp (annual mean),15.289717503236119
100 hPa equatorial temp (annual cycle strength),1.6689598259361844
100 hPa 10Sto10N temp (annual mean),15.478782616610857
100 hPa 10Sto10N temp (annual cycle strength),1.614943422793658
70 hPa 10Sto10N wv (annual mean),5.748239321012629
Summary,12.025357747333425
RMS error: tropical Age of Air,-10000.0
RMS error: NH midlatitude Age of Air,-10000.0

and the plots now look identical as well: https://github.com/NCAS-CMS/NCAS-Useful-Documentation/tree/master/autoassess_review_results/stratosphere_AERmonZ/plots/aa_strato/autoassess_strato_test_1/HadGEM3-GC31-LL_vs_UKESM1-0-LL/stratosphere

Note that since I don't have ERA-I AERmonZ data I couldn't produce the last plot (t100_vs_q70) but the metrics plot and all else in there

:beer:

valeriupredoi commented 5 years ago

oh and I forgot, I used the cosine weights calculation, indeed it does the job, see https://github.com/ESMValGroup/ESMValTool/pull/1192/commits/d5c0adc2a66fe80ea0c3d1a52584a1c0b13373be

alistairsellar commented 5 years ago

Fantastic - great work!

The t100 vs q70 plot that you worked so hard on is missing from the folder above. Can you run that one again?

And could you include the obs file argument when you make Stratosphere_Metrics.png? This makes obs ranges appear on the plot. They were there in an earlier version you made.

valeriupredoi commented 5 years ago

dude, I don't have obs for AERmonZ

valeriupredoi commented 5 years ago

updated documentation as well: https://github.com/ESMValGroup/ESMValTool/blob/version2_development_autoassess_recipes_documentation/doc/sphinx/source/recipes/recipe_autoassess_stratosphere.rst

alistairsellar commented 5 years ago

I don't have obs for AERmonZ

I get it - so the obs must be the same shape as the mode data. And you're going to open an issue to create an AERmonZ (i.e. zonal mean) variant of ERA-interim.

For the obs ranges (or more precisely, the acceptable ranges according to expert judgement) in the metrics plot, these need to be specified via an external csv file. You're going to open another issue to enable to code to take these files as input. There should be an argument in the original metrics plot routine for these "obs" files. The input files are here for stratosphere and land_surface: https://code.metoffice.gov.uk/trac/cma/browser/autoassess/trunk/autoassess/assessment_areas/stratosphere/assessment_plot_obs/stratosphere_obs.csv https://code.metoffice.gov.uk/trac/cma/browser/autoassess/trunk/autoassess/assessment_areas/land_surface/assessment_plot_obs/land_surface_obs.csv

alistairsellar commented 5 years ago

Setting aside the two issues above (to be deferred to later work), I'm happy that the stratosphere metrics / plots are scientifically correct.

valeriupredoi commented 5 years ago

@alistairsellar and myself concluded that the review for autoassess/stratosphere is virtually finished (yesh! beers to us :beer: ). There are however a few things that we need to do next:

But publication (in Paper Part 2) and celebration can now ensue :grin:

valeriupredoi commented 5 years ago

associated issues open and the review has now been signed off by @alistairsellar and myself so am gonna close this. Well done to us :beer: