search

judithberner / climpred_CESM1_S2S

MIT License
3 stars 1 forks source link

contribution guide #10

Closed aaronspring closed 3 years ago

aaronspring commented 3 years ago

edit: was initial RPSS example. Then only contrib guide

after installing https://github.com/pangeo-data/climpred/pull/566 and https://github.com/xarray-contrib/xskillscore/pull/277

Todo:

judithberner commented 3 years ago

So we are working on some issues around weekly averages, which is in our court. As we had discussed the skill is so low, that we don't take the weekly average over the daily metrics (e.g. rmse, rps, ...), but compute the metric over weekly anomalies (e.g. can the model predict a warm week from a cold week). So some of our input zarr-files (on NCAR machine) might not be correct.

Also - just to be clear - we want to keep lat/lon dependence in the metrics as long as possible, so that we can show maps (as you do). The lat/lon averaging is done just before making line plots of the metrics.

Once we have solved this, here are some sanity checks. The RPSS for T2m for ECMWF week 1-2 is around 0.01 (reference to follow), so it's positive but very small. For RPS I would expect a larger number > 1.

“The approximate relationship between RPSS and correlation being such that a RPSS value of 0.1 corresponds to a correlation of about 0.44 (Tippett et al. 2010).”

aaronspring commented 3 years ago

So we are working on some issues around weekly averages, which is in our court. As we had discussed the skill is so low, that we don't take the weekly average over the daily metrics (e.g. rmse, rps, ...), but compute the metric over weekly anomalies (e.g. can the model predict a warm week from a cold week). So some of our input zarr-files (on NCAR machine) might not be correct.

might be related to https://github.com/judithberner/climpred_CESM1_S2S/issues/11 says @abjaye

aaronspring commented 3 years ago

on capser I have a weekly verif example. skill looks different.

aaronspring commented 3 years ago

wasnt clever of me to overwrite parts of the rpss notebook. maybe I just merge the contributing guide then.

judithberner commented 3 years ago

Just to be clear: our latest notebook (with issue 11) is here: https://github.com/judithberner/climpred_CESM1_S2S/blob/main/all_scores_rpss.ipynb

rpss.ipynb is an older version.

aaronspring commented 3 years ago

thanks. then I would cleanup my notebook here and overwrite the previous rpss notebook once the machine is back online