reconciliate perfect-model with hindcast approach

[aaronspring]

maybe we can combine the two approaches:

• perfect-model has no reference, therefore we compare model with itself

  • input: ds, [control (only needed to bootstrap uninitialized), should be refactored out of this core function]
  • ds.sel(member=0) is the control member

• hindcast: compare against reference

  • input: dple, reference

• proposals to combine: either ... or

  • adapt perfect_model:
    1. split reference member from ds into reference, keep remaining as ds
    2. compute_reference
    3. loop over all members as reference, mean results
  • adapt compute_reference:
    1. combine reference as member 0
    2. perfect_model(ds,..., comparison='m2c')

[aaronspring]

the underlying conflict is the different ways how metrics work and are applied:

• ACC: computes the correlation between obs and fct over all init, over all 'member' fails because reference is identical for all members and therefore skill = 0
• MSE (and all others) compute a variance between obs and fct in member first and then mean over all init [Goddard, 2013] and all the perfect-model studies use this approach: [Griffies, 1997; Pohlmann, 2004; Seferian, 2018]

Bushuk [2018] uses both MSE and ACC and uses different approaches as we do:

• operational metrics: our compute_reference/hindcast

  • example MSE:

  • example ACC:

• PM metrics: our compute_perfect_model (using the supervector approach)

  • example MSE:

  • example ACC:

Either ... or:

• we want methodology of how we treat metrics in a given experiment consistent (current state and my favorite)
• we want how we define metrics consistent at the cost of different methodology

Therefore, I cannot see a way how to reconciliate the two approaches here. I am in favor of letting as is.

[aaronspring]

another thing really needed to reconciliate is the get the PM and hindcast approach on the same input dimensions before entering metric(forecast, reference). this is currently not the case.

one approach would be to create a reference with coords lead and init out of time and then concat this a member 0 with the hindcast and then supply this into compute_perfect_model(comparison=e2c) as e2c compares the ensemble mean to member 0.

[aaronspring]

my big problem with all this is that unless we reconciliate compute_perfect_model with compute_reference we will also need to implement how we use all the metrics twice. I dont see a way right now how to implement a metric which is not in xskillscore into compute_reference.

[bradyrx]

I agree with you totally. I think there might be a way to modify compute_reference to handle probabilistic and skill scores. I will give it some thought unless you are going to suggest a method in the upcoming PR.

[aaronspring]

I think I will kick out control, running and reference_period from compute_perfect_model. With the bootstrapping as reference forecasts to identify significance, this is not needed anymore. This will simplify the API of compute_perfect_model.

[bradyrx]

Addressed in https://github.com/bradyrx/climpred/pull/128