Closed ngupta23 closed 3 years ago
FB Prophet now uses the same rolling window cross validation folds strategy that all other models use. This has been fixed in the 0.0.26 version. This can be tested by re-installing auto_ts.
Please try again using the latest version: "pip install auto_ts --ignore-installed --no-cache-dir"
That should upgrade your version and hopefully fixes it.
The number of folds and the content of these folds is the same for all models now. Fixed - just upgrade to the latest version by: "pip install auto_ts --ignore-installed --no-cache-dir"
Updates made for Prophet till https://github.com/AutoViML/Auto_TS/commit/8097292415b1c9e03f500feaf05ecd76c7609789 were predominantly consistent with the other models in that they gave the same number of cv splits and number of observations in each split (for the most part). This was tested on some
monthly
datasets only.However, there seemed to be some issue with it for other datasets as per @AutoViML. Hence the implementation was changed from this release.
However, this leads to 2 issues
The number of CV folds for Prophet does not match that for other models now. One such example can be found here: https://github.com/AutoViML/Auto_TS/blob/a162b1a4297281c991f6cd76d8482b2393e0add5/auto_ts/test/test_auto_ts.py#L771
The number of observations in each fold does not match the forecast horizon as can be seen here where horizon is 8 but number of observations in the validation set of each fold is 13. https://github.com/AutoViML/Auto_TS/blob/a162b1a4297281c991f6cd76d8482b2393e0add5/auto_ts/test/test_auto_ts.py#L68
Ultimately, these 2 bugs will need to be fixed to get an unbiased comparison of all models in the AutoML flow.