search

ACCESS-NRI / accessdev-Trac-archive

Archive accessdev Trac contents as issues
Apache License 2.0
0 stars 0 forks source link

Assess OPS and VAR of G3 N1024 Terra trial using Varstats #364

Open penguian opened 6 years ago

penguian commented 6 years ago

keyword_Varstats_OPS_VAR_G3 | by jtl548@nci.org.au

It appears the APS3 ACCESS-G N1024 trial that ran on Terra did not perform as expected. In this ticket Varstats will be used to the assess the OPS and VAR aspects of the trial.

Issue migrated from trac:364 at 2024-01-31 18:34:15 +1100

penguian commented 6 years ago

@peter.steinle@bom.gov.au commented

Corresponding Gitlab Issue: https://hpc-gitlab.bom.gov.au/ACCESS/APS3/GlobalSuite/issues/73

penguian commented 6 years ago

@jin.lee@bom.gov.au _uploaded file ps39_m400_trial_ps39_m400_trial_copy_var_iasi_ma_countmean_AveVals_i0_l1_s3_ch164--263.png (73.2 KiB)_

ps39_m400_trial_ps39_m400_trial_copy_var_iasi_ma_mean_AveVals_i0_l1_s3_ch164--263.png

penguian commented 6 years ago

@jin.lee@bom.gov.au commented

VAR stats files

2 trials are compared. The source of their data are,

Varstats

Varstats expects 2 trials - experiment and control - to be run over the same period. For the current comparison this is not the case. To get around this atypical usage dummy runs were created - copies of experiment and control.

Results

  1. For many of the observation types more observations are used in G3 than in UKMO global: see "GPSRO_countmean_AveVals", "iasi_ma_countmean_AveVals", for example. Exceptions are Satwind, Scatwind, Surface,

  2. For most observation types the mean of the first-guess departure (o-b) is similar in both trials (perhaps G3 is slightly worse). However G3 has higher standard deviation of o-b for most observation types. Presumably this is because the forecasts from the UKMO global is better than those from G3.

  3. Comparing the values of observation penalty functions for a number of observation types the numbers for G3 are larger than for those of UKMO. Most likely because more observations are used in G3 than in UKMO: e.g. see "GPSRO_penalty_ts_penalties", "iasi_mb_penalty_ts_penalties"

  4. We know there were problems with the assimilation of AMV's in G3. Looking at the plots for the Satwind observation type, a. Contrary to other observation types less Satwind observations are used in G3 than in UKMO: ~24000 for G3 compared to ~40000 on average (see "satwind_U_countmean_AveVals") b. Comparison of the mean values of of o-b shows mixed result

penguian commented 6 years ago

fs5892 commented

I am not convinced the biases are consistent enough. The biases in the tropospheric temperature sounding channels for the hyperspectral sounders, and also AMSU-4 is quite a lot bigger in G3. Also the bias against Sonde for temperature is larger throughout the troposphere. The lower level humidity also looks more biased in G3 - some AMSU-B/MHS show this, but also the sonde bias looks quite different.

Sonde wind biases are also different, but this could be explained by the lack of AMV data.

Also, we shouldn't be assimilating SSMIS until May 2018. This needs to be switched off. I did mention this previously. http://logan.bom.gov.au:8011/nwp-wg/wiki/SSMISAvgDataCheck.

We should bear in mind this trial is not running with recommended SatRad coefficients (they were only fixed yesterday, so clearly it wasn't) - you can see this because SSMIS 2&5 are still on and have been switched off in the new version. I don't think this is a major issue - the only changes are switching off channels that might cause problems under certain circumstances which are unlikely to be met in a given short period.

penguian commented 6 years ago

fs5892 commented

I've been looking at this again in a new light having just checked the new G3 setup. I think I was a bit over-zealous in my interpretation of the differences in the means - they are bigger for G3, but very small overall, so probably not too big a concern for now.

A comment on the observation counts, following conversation with Jin: Although there are a few mysteries, actually the observation counts are pretty similar if you compare G3-glm vs MO-glu.