search

ACCESS-NRI / accessdev-Trac-archive

Archive accessdev Trac contents as issues
Apache License 2.0
0 stars 0 forks source link

Assess possible degradation of TC forecasts by ACCESS-G caused by not assimilating some observation types #370

Open penguian opened 5 years ago

penguian commented 5 years ago

keyword_tropical_cyclones_global_NWP_observation_impact type_non-CMIP6 | by jtl548@nci.org.au

Currently operational ACCESS-G2 and the next upgrade, ACCESS-G3 do not assimilate a number of observation types which are well established in UKMO and other operational NWP centres. The bulk of these unused observation types is from various satellite instruments. It is commonly thought that these unassimilated observations result in poorer observation coverage for the Bureau's global NWP. Of particular interest is the impact of this poorer coverage for the tropics.

In this experiment we will try to test whether the current incomplete observation coverage has any impact on the quality of ACCESS global 4DVar analyses and as a consequence the forecast model's ability to accurately predict tracks of tropical cyclones.

Issue migrated from trac:370 at 2024-01-31 18:34:57 +1100

penguian commented 5 years ago

@jin.lee@bom.gov.au commented

Ticket Details

penguian commented 5 years ago

@jin.lee@bom.gov.au commented

Hi Andy,

I think I now have correct setups for the control and the experiment. When you have time do you mind casting a glance over the suites to make sure there's no obvious mistakes? The suites are,

control: u-bl688 experiment: u-bm460

Thanks.

penguian commented 5 years ago

as2291 set owner to jtl548

penguian commented 5 years ago

as2291 commented

This all looks ok. As discussed, I think it's worthwhile switching off the analysis update task as it doesn't do anything useful and will save resources.

penguian commented 4 years ago

@jin.lee@bom.gov.au set type to CMIP (DECK/hist)

penguian commented 4 years ago

@jin.lee@bom.gov.au commented

For a while there was some uncertainty whether the results of the experiment was credible. After comparing with similar trial that Fiona did earlier on UKMO XC I am now confident with the conclusion that assimilating those extra observations that UKMO uses but ACCESS-G3 did not (at the time of this trial) has no impact on the forecast quality. Furthermore the verification of TC tracks shows that those extra observations has no impact on tropical cyclone tracks.

These conclusions are only valid for the set of configurations used in this trial: N320 UM, N108/N216 PFM, channel selection, stationlist, thinning distance, etc. etc. As to the question, whether the result depends on one or more of the configurations, it is to be investigated in a future study.

penguian commented 4 years ago

@jin.lee@bom.gov.au changed type from CMIP (DECK/hist) to non-CMIP6

penguian commented 4 years ago

fs5892 commented

Jin, can you link results plots from my trials?

I am still very confused by this result, because all of these extra components gave a small amount of impact when added at the Met Office (I was coordinating the package trial where a lot of them went in). If there was only neutral impact on scores, there was always an improvement in fit to other obs. There is a reasonable number of missing obs in this package that you ran - you would expect to see something.

Did you run Boite Noire? It would be interesting to see the results of that.

penguian commented 4 years ago

@jin.lee@bom.gov.au commented

Fiona,

The details are in Ticket Details (wikipage linked from this ticket - see the first post).

All the results are under the heading, Results (The version of Trac on accessdev is so old that it doesn't support anchors. So I can't create a direct link from here. Look for the heading).

The result from your earlier trial that ran on UKMO XC is under the heading, Comparison with similar data denial experiment conducted by Fiona Smith. Within this heading there are 2 sub-headings,

As to Boite Noire, Brett Candy when he was sending me the ARD's from your trial didn't include log files. So unfortunately I don't have Boite Noire results.

I am confused as much as you are. I would be very interested in your thoughts on the result.

penguian commented 4 years ago

@jin.lee@bom.gov.au commented

Hi Fiona and others,

Have you had chance to look at the results more closely? I'm planning a paper but before I start writing the first draft I would like to hear some critical reviews of the experiment and its results.

penguian commented 4 years ago

@peter.steinle@bom.gov.au commented

Am surprised by how little impact the "no hyperspectral sounder" sanity check experiment has at 250 in tropics & southern hemisphere. Realize it comes up as a significant change in verification summary, but when looks at the actual scores against sonde it is much less than I would have expected. Not quite sure of what to make of it

penguian commented 4 years ago

@peter.steinle@bom.gov.au changed _comment0 which not transferred by tractive

penguian commented 4 years ago

@jin.lee@bom.gov.au commented

Replying to ACCESS-NRI/accessdev-Trac-archive#370 (comment:10):

Am surprised by how little impact the "no hyperspectral sounder" sanity check experiment has at 250 in tropics & southern hemisphere. Realize it comes up as a significant change in verification summary, but when looks at the actual scores against sonde it is much less than I would have expected. Not quite sure of what to make of it UKMO did a denial experiment where they removed all IR observations. The twist to the story is that the impact was more noticeable when verified against ECMWF analyses. But when the same trial was verified against observations they got results comparable to mine. I added their score card and email reply from Stuart Newman in the "no hyperspectral sounder" sanity check wikipage (wiki:ticket/370/TicketDetails/SanityCheck). Note that their trial was a complete denial of IR observations whereas I only removed hyperspectral sounder data. So they saw slightly bigger impact.

penguian commented 4 years ago

@jin.lee@bom.gov.au commented

Hi all,

The VarStats plots now make sense. As expected the number of observations used in the experiment (i.e. the denial run) decreased and the experiment shows worse o-b. I added some more comments under the heading, Results and the subheading, Overall verification for those who want to scrutinise the results.