Closed aaronspring closed 5 years ago
the underlying conflict is the different ways how metrics work and are applied:
obs
and fct
over all init
, over all 'member' fails because reference is identical for all members and therefore skill = 0
obs
and fct
in member
first and then mean over all init
[Goddard, 2013]
Bushuk [2018] uses both MSE and ACC and uses different approaches as we do:
operational metrics: our compute_reference/hindcast
example MSE:
example ACC:
PM metrics: our compute_perfect_model (using the supervector approach)
example MSE:
example ACC:
Either ... or:
Therefore, I cannot see a way how to reconciliate the two approaches here. I am in favor of letting as is.
another thing really needed to reconciliate is the get the PM and hindcast approach on the same input dimensions before entering metric(forecast, reference)
. this is currently not the case.
one approach would be to create a reference with coords lead
and init
out of time
and then concat this a member 0 with the hindcast and then supply this into compute_perfect_model(comparison=e2c)
as e2c
compares the ensemble mean to member 0.
my big problem with all this is that unless we reconciliate compute_perfect_model
with compute_reference
we will also need to implement how we use all the metrics twice. I dont see a way right now how to implement a metric which is not in xskillscore
into compute_reference
.
I agree with you totally. I think there might be a way to modify compute_reference
to handle probabilistic and skill scores. I will give it some thought unless you are going to suggest a method in the upcoming PR.
I think I will kick out control
, running
and reference_period
from compute_perfect_model
. With the bootstrapping as reference forecasts to identify significance, this is not needed anymore.
This will simplify the API of compute_perfect_model
.
Addressed in https://github.com/bradyrx/climpred/pull/128
maybe we can combine the two approaches:
perfect-model has no reference, therefore we compare model with itself
hindcast: compare against reference
proposals to combine: either ... or