owrior
commented
2 years ago Currently tests for sklearn are done with only the next day which can give a false impression of performance. Therefore I think this should also predict in to the future based on calculated values as is done in the actual forecasting. This can also be used for the prophet calculation.
This will also mean that performance estimates can be given based on how far in the future you are predicting giving an idea of the dropoff.
This will require:
- A new function for predicting sklearn models in to the future.
- New functionality for validation in to the future based on the time windows. Apply this to prophet as well.
Prophet comes with built in cross validation functionality. However this is handled differently to the traditional method used and is not compatible with the other ML metrics which can make the estimated parameters misleading.
To combat this currently a very simple method of creating time windows and calculating metrics based on train set performance - which is very far from ideal.
Either new functionality for the sklearn validation should be used which matches the prophet methodology more closely (as this is most likely more suitable for timeseries) or a proper K fold implementation matching the sklearn method for prophet should be created.