search

judithberner / climpred_CESM1_S2S

MIT License
3 stars 1 forks source link

Add RPSS to climpred #7

Closed judithberner closed 3 years ago

judithberner commented 3 years ago

@aaronspring Thanks for your work wrt the RPS. What we really want is the RPSS. RPSS= 1- RPS_experiment/RPS_reference.

I tried to use persistence as a reference, but the code failed: https://github.com/judithberner/climpred_CESM1_S2S/blob/main/rpss.ipynb (Search for "Aaron"). You should be able to run the code using the intermediary files from Abby's directory after launching the dask servers via https://github.com/judithberner/climpred_CESM1_S2S/blob/main/0.0_launch_cluster.ipynb and pasting the scheduler into the notebook.

Error message: "ValueError: 'member' not found in array dimensions ('time', 'lat', 'lon')"

Note that we are interested in the skill of "weekly averaged" anomalies, which is different from the "weekly averaged skill of daily anomalies" (and really gets at the tercile forecasts). While the error message suggests something different, I assume we need to do the weekly averaging of the persistence to be consistent.

I think this shouldn't necessarily go into "climpred" skill=hindcast2_djf.verify(metric="rps",alignment="maximize",comparison=comp,dim=dim,reference="persistence",category_edges=bins)

but maybe we can change hindcast2_djf.verify(metric="rps",...) to accept an externally computed reference?

Thanks! @abjaye

aaronspring commented 3 years ago

RPSS= 1- RPS_experiment/RPS_reference.

Can we do this step by step?

  1. RPS_experiment
  2. RPS_reference
  3. RPSS by formula

Now we disable the persistence forecast in climpred for all probabilistic forecasts because they (often) require a member dimension and a persistence forecast is only one member. We came up with this. Maybe they should be a persistence forecast for RPS. Do you have a reference or description how to compute it? Whats the persistence or climatology assumption for probabilistic? Maybe All categories as probable as a Gaussian distribution?

aaronspring commented 3 years ago

Once we figure out the flow, adding rpss should be straightforward

judithberner commented 3 years ago

Yes - I think three steps is they way to go. The standard for RPSS is climatology, not persistence. But they are both deterministic, i.e. don't have a member dimension (I guess it you could define a persistence for each member). I will close this issue and transfer to climpred. OK?

aaronspring commented 3 years ago

yep. please close here. thanks for the transfer.