search

business-science / modeltime.resample

Resampling Tools for Time Series Forecasting with Modeltime
https://business-science.github.io/modeltime.resample/
Other
19 stars 5 forks source link

Could you clarify how the rolling root mean squared error (RMSE) is calculated? #9

Open wave1122 opened 2 years ago

wave1122 commented 2 years ago

Suppose that I have two slices, say [1, 2, 3, 4] and [2, 3, 4, 5]; and I want to make a two-period-ahead forecast in each slice. For slice 1, I used the first two observations [1, 2] to train a model which will then be used to forecast the last two observations [3, 4]. Similarly, for slice 2, I used [2, 3] to train a model which will then be used to forecast [4, 5]. Let's assume that I obtain [3.5, 4.5] as the forecasts of [3, 4] in slide 1 and [4.1, 5.1] as the forecasts of [4, 5] in slide 2.

I then proceed to calculate the RMSE for slide 1 as $sqrt[( (3.5 - 3)^2 + (4.5 - 4)^2 ) / 2] $ and the RMSE for slide 2 as $sqrt[( (4.1 - 4)^2 + (5.1 - 5)^2 ) / 2]$. Is this how you calculate the rolling RMSe in modeltime?

Thank you!

Shafi2016 commented 2 years ago

I am working on a similar issue, I think it will depend on the "resamples" strategy. If you use tscv from timetk or https://rsample.tidymodels.org/reference/slide-resampling.html#examples-1 will give us different results. modeltime_fit_resamples() estimate the RMSE for each slice.