search

facebook / prophet

Tool for producing high quality forecasts for time series data that has multiple seasonality with linear or non-linear growth.
https://facebook.github.io/prophet
MIT License
18.42k stars 4.53k forks source link

Unexpected yearly seasonality results using known test cases #463

Closed Mjboothaus closed 6 years ago

Mjboothaus commented 6 years ago

G'day

Thanks again for open-sourcing fbprophet.

I wanted to do some testing of the library using synthetic data, before using it on real data. Attached to this issue is a zipped Jupyter notebook that provides the analysis that I conducted. Prophet_Testing.zip

I found that while for a number of test cases (by no means an exhaustive list) the library appears to perform very well, there is one case here (Case 7) where it does not reproduce the season trend.

Any thoughts on these findings would be appreciated. For example, is it a bug? Or is it a limitation of the underlying maths/fitting procedure.

Cheers, Michael

Summary of the analysis / Jupyter notebook:

  1. The versions of Python libraries I am using are captured in cell [1].

  2. The test cases are all composed of a trend which is either constant (as in Case 1) or logistic growth combined with a (sinusoidal) seasonal component for Cases 3-7.

  3. The final chart in each case compares the input (orange line) to the forecast (blue line) and a measure of the difference of the two seasonal curves is reported (square root of sum squared differences).

  4. In test Cases 1-6, the forecasts do a pretty reasonable job of reproducing the input trend and seasonal pattern.

  5. In Case 1 there is exact agreement.

  6. In Case 2 there is good overall agreement but the predicted seasonal trend (which should be zero) has a particular shape although the absolute deviations from zero are not large (especially compared to the overall value of the function).

  7. Cases 3-6 show pleasing results - with the seasonal component well reproduced.

  8. However in Case 7 (as seen in the 2nd last cell in the Notebook) the forecast fails to reproduce the input seasonality and overall forecast. I imagine that this behaviour will also be observed for a range of other test cases, but I wanted to illustrate the behaviour with this one.

bletham commented 6 years ago

Prophet models seasonality with a truncated Fourier series. The truncation behaves as a low-pass filter that prevents the seasonality model from capturing high-frequency changes. In case 7, the frequency was too high for the default number of Fourier terms used. That default is 10, and you can increase it using the yearly.seasonality input like so:

model = Prophet(daily_seasonality=False, weekly_seasonality=False, yearly_seasonality=15)

Increasing to 15 from the default 10 allows it to fit Case 7 nicely.

This relates to #409, which is that we need to improve the documentation around Fourier order so that it is easier to understand what the effect of this number is and when you might need to increase it.

Mjboothaus commented 6 years ago

Thanks so much for your reply!

I was wondering if it is possible to automatically "optimise" the number of Fourier terms used (as part of the overall fitting). I guess I assumed this number was being varied when the 'auto' setting is used for yearly_seasonality. I take your point in #409 that increasing this number too far will lead to over-fitting.

If you'd like some proof reading of the additional documentation / examples then feel welcome to ask.

bletham commented 6 years ago

In principle one could use cross validation to select the right number of terms - however because the cross-validation does not provide independent estimates of error, I think you'd have to have a pretty long history to be able to successfully select multiple parameters via cross-validation.