search

jejjohnson / oceanbench

OceanBench - SSH edition
https://jejjohnson.github.io/oceanbench/
MIT License
17 stars 2 forks source link

Metrics (Real Space) #23

Closed jejjohnson closed 1 year ago

jejjohnson commented 1 year ago

We want some standard metrics that we can use to evaluate our performance. The metrics referenced in this issue will be in real space, i.e. directly comparing variable for variable.

Previous Work

xskillscore

They have a suite of correlation-based, distance-based, and probabilistic-based metrics.

They also have a nice API whereby we just say we want 

da_ref = ...
da_target = ...
# skill score for every coord in time
# (lat,lon,time) -> (lat,lon)
r = xs.pearson_r(da_ref, da_target, dim="time")

# skill score for every coord in space
# (lat,lon,time) -> (time)
r = xs.pearson_r(da_ref, da_target, dim=["lat", "lon"])

# everyones favourite MAPE
mape = xs.mape(da_ref, da_target, dim=["lat", "lon"])

We can offset a lot of the hard work for calculating the metrics. In fact, we could even try and contribute to this package if we think it's worth it.

skillmetrics

This package has a few nice plots, e.g. The Taylor Diagram.

jejjohnson commented 1 year ago

Closing due to migration to oceantools. See issue jejjohnson/ocn-tools/issues/3