search

AutoViML / Auto_TS

Automatically build ARIMA, SARIMAX, VAR, FB Prophet and XGBoost Models on Time Series data sets with a Single Line of Code. Created by Ram Seshadri. Collaborators welcome.
Apache License 2.0
729 stars 114 forks source link

Biased estimation for Prophet #25

Closed ngupta23 closed 3 years ago

ngupta23 commented 3 years ago

Updates made for Prophet till https://github.com/AutoViML/Auto_TS/commit/8097292415b1c9e03f500feaf05ecd76c7609789 were predominantly consistent with the other models in that they gave the same number of cv splits and number of observations in each split (for the most part). This was tested on some monthly datasets only.

However, there seemed to be some issue with it for other datasets as per @AutoViML. Hence the implementation was changed from this release.

However, this leads to 2 issues

  1. The number of CV folds for Prophet does not match that for other models now. One such example can be found here: https://github.com/AutoViML/Auto_TS/blob/a162b1a4297281c991f6cd76d8482b2393e0add5/auto_ts/test/test_auto_ts.py#L771

  2. The number of observations in each fold does not match the forecast horizon as can be seen here where horizon is 8 but number of observations in the validation set of each fold is 13. https://github.com/AutoViML/Auto_TS/blob/a162b1a4297281c991f6cd76d8482b2393e0add5/auto_ts/test/test_auto_ts.py#L68

Ultimately, these 2 bugs will need to be fixed to get an unbiased comparison of all models in the AutoML flow.

AutoViML commented 3 years ago

FB Prophet now uses the same rolling window cross validation folds strategy that all other models use. This has been fixed in the 0.0.26 version. This can be tested by re-installing auto_ts.

Please try again using the latest version: "pip install auto_ts --ignore-installed --no-cache-dir"

That should upgrade your version and hopefully fixes it.

AutoViML commented 3 years ago

The number of folds and the content of these folds is the same for all models now. Fixed - just upgrade to the latest version by: "pip install auto_ts --ignore-installed --no-cache-dir"