search

iri-pycpt / pycpt

package joining pycpt stuff
3 stars 1 forks source link

Implement double cross-validation #26

Open aaron-kaplan opened 1 year ago

aaron-kaplan commented 1 year ago

and stop faking dates to work around CPT's refusal to score in-sample hindcasts.

simon2212 commented 1 year ago

I'm new to this forum, so am quite possibly missing something. I just see 2 entries - the implementation request, and a comment about fake dates. CPT purposely makes it hard to score in-sample hindcasts because of the artificially high skill estimates you get. However, if you have a sample size of n, you could easily get CPT to produce n probabilistic forecasts for those n years by training the model, and then using the forecast option (451) by setting the forecast file (option 3) to be the same as the X file (option 1), and then setting the start-at date (option 223) to the X start at date, and the number of forecasts (option 9) to n. To get scores, you would need to save output, and read it back in using PFV.