search

dtcenter / MET

Model Evaluation Tools
https://dtcenter.org/community-code/model-evaluation-tools-met
Apache License 2.0
78 stars 24 forks source link

Enhance Stat-Analysis to support the direct comparison of the performance of two different models. #61

Open dwfncar opened 13 years ago

dwfncar commented 13 years ago

Most of the DTC verification projects involve inter-comparison of two or more models. New Stat-Analysis jobs could be defined to aid in those comparisons. However, those jobs would need to be clearly defined. [MET-61] created by johnhg

dwfncar commented 9 years ago

Hi Tressa (and Barb),

Here is code to do bootstrapping (IID percentile and BCa, and Circular block bootstrap with percentile). I run a few different types of comparisons (e.g., taking the two means first and then subtracting them vs taking the mean of the loss differential series), just as a sanity check.

I find that the BCa method often fails with an error, so you may want to run it without (default does everything).

Let me know if you have any questions about the code, or how to use it. I couldn't get R to do exactly what I wanted, so I wrote a print method function for it, which should help in terms of deciphering the results.

Just give it two difference series, and let it rip. For example, say I have F1 (forecast 1), F2 (forecast 2) and O (observation), where each is a time series.

bootcompare( e1 = F1 - O, e2 = F2 - O, type = "perc" )

We should also think about whether or not we want to do any testing with the other tests that Del Sole and Tippitt discuss (like the ST, etc.). My feeling is that it might be overkill at this point, but we could always do something with them later. There is no way to directly compare them with the other loss functions, and it seems like they are not tests that we've employed in our various reports (maybe we should start?).

Eric by johnhg

dwfncar commented 7 years ago

This would be very useful for EMC people working on HPCs and not wanting to push into METViewer. Requested by Tom Hamill at PSD as well. Would use event equalization logic supported in METViewer. by jensen

dwfncar commented 7 years ago

Job number = 275200 by jensen

dwfncar commented 6 years ago

We need -event_equal logic.
We need some way of defining what columns define the pair-wise difference.
We need the ability to bootstrap aggregations and do pair-wise differences along the way.
We should also do bootstrapping during the summary job and support pairwise differences there (i.e. mean of daily RMSE differences between 2 models).

by johnhg

lisagoodrich commented 5 years ago

Original JIRA location: https://sdg.rap.ucar.edu/jira/browse/MET-61?filter=12903

Remove the .txt after downloading file. github can't save .R extensions bootcompare.R.txt