Explanation for 100% accuracy for all historical records for January

[dhanashreearole]

Hello,

I had dataset at monthly level from Jan 2012 through August 2018. I observed that January's had highest variance. So I segmented the data into jan dataset and nonjan dataset and ran the predictions with reciprocal square transformation. The result for Jan is:

The result for nonJan is:

Can somebody justify how prophet goes ahead and gives 100% accuracy for all the records of January? This prediction has been done at annual level with YS (start of year) as frequency. I tried other transformations like log, reciprocal and root reciprocal per Tukey Ladder of Power and it has same behavior of 100% accuracy for every january, however it drastically show different predicition for January 2019 depending on which transformation is used. Reciprocal square is most acceptable since it's value is in the range and comparable with previous January's.

Please chime in your thoughts.

Thanks, Dhanashree

[dhanashreearole]

Also, I was wondering why the lower and upper forecast are almost similar to forecast for January predictions?

[bletham]

Is it correct that you are computing error here by (1) fit the model, (2) make predictions on the history, and (3) compare those predictions to the actual values?

In that case, the error that you are measuring here is the "fit" error (in-sample). What you really care about is the prediction error (out-of-sample). Meaning, here you are measuring how good Prophet is at forecasting data that it has already seen - it's really good! No surprise, the model already knows the true values, and so it should do a really good job. What you want to do is ask the model to make predictions at dates that it hasn't yet seen. You do this by fitting the model to a subset of the data (e.g., 2012-2104) and then asking the model to make predictions for later data (2015-2018). See https://facebook.github.io/prophet/docs/diagnostics.html for some built-in tools for doing that sort of analysis.

If you do this, I think you'll see that the prediction error for Jan is quite high. What's happening here is that the model is just fitting a line that passes perfectly through the Jan data, which means it's overfitting badly. This is because it has only 1 datapoint per year for 7 years, which isn't really enough data for it to figure out what the actual trend should be other than just matching the data.

For the not-Jan data, there is enough data for it to actually learn some yearly seasonal effects, and it probably is doing a much better job. (fit error means that it isn't overfitting as badly, and prediction error should be a lot lower).

I don't think that you will be able to get much out of separating out Jan, and I suspect there is a better way. If you post a plot of the model forecast to all data (that is, including Jan), I could take a look and see what you mean by high variability in Jan and see if there is a way to handle it that is better suited for Prophet.

[dhanashreearole]

Thanks for the explanation. Here is the prediction for entire dataset with reciprocal square transformation, without segmenting January months.

Here is the prediction with log transformation:

Attached are the outputs.

id_battery_jan_12_66_05_16_monthly_ben_log.xlsx id_battery_jan_12_66_05_16_monthly_reciprocalsquare_ben.xlsx

If you can suggest a way to improve the accuracy of Jan 2018 which is currently 61%, it would be good advice as I am guessing that low accuracy is going to predict Jan 2019 poorly as well.

Thanks!!!

[dhanashreearole]

Ben, I was able to improve the accuracy by traversing with the transformation stepwise from 1.125 to 1.5 with 1.2 as the optimal value that gives 74% accuracy for Jan 2018. It is much better.

But as always I am open to any other advice you may have :)

Thanks!

[bletham]

I see, this is a challenging behavior for Prophet to capture.

Basically what is happening here is that the yearly seasonality effect in January is changing. In 2013 and 2014 there were huge drops, then in 2015-2017 smaller drops. Finally in 2018 there was basically no drop.

One thing that is happening is the size of the seasonal effect overall is shrinking as the trend continues downward - this is indicative of multiplicative seasonality, and I'd expect you'd have a little better performance if you turn that on: https://facebook.github.io/prophet/docs/multiplicative_seasonality.html

But otherwise, there isn't really any way for the model to know why Jan 2018 was so different than Jan 2017. Should we expect 2019 to be like 2018 or 2017? The model doesn't really know and so what it produces is an effect that is essentially the average of all of the drops seen from 2012-2018.

If you do have some extra knowledge about why the January effect has been changing, then you'd need to incorporate that knowledge into an extra regressor (https://facebook.github.io/prophet/docs/seasonality,_holiday_effects,_and_regressors.html#additional-regressors) that for instance would have a higher value in 2018 than 2017, and for 2019 would do what you with your extra knowledge know is the right trend in January effects. But otherwise the model doesn't really have any way to figure out why the Jan effect is changing.

[dhanashreearole]

Thanks Ben! It helped improve the accuracy by 2% for Jan 2018, which I am sure will reflect in Jan 2019 prediction as well.

I expect actual call volume to be in the range of lower and upper forecast for historical call volume. However, I see some odd ducks out. Can you explain why sometimes that happens?

For example, for Jan 2018, 842 is not in the range of 929 and 1346.

Is it directly related to accuracy level, if accuracy level is well above 90% then actual will be in the range, else it is not guaranteed?

Thanks!

[bletham]

By default the uncertainty interval given is an 80% interval, meaning that if the model assumptions were all true, there would be an 80% chance that the intervals cover the points. However, the model assumptions are certainly not all accurate. For instance, the model assumes i.i.d. Gaussian noise, which is not consistent with the outliers in the figure. See https://facebook.github.io/prophet/docs/uncertainty_intervals.html for more about the assumptions behind the uncertainty intervals. So, the actual coverage (how many points fall within the uncertainty intervals) will depend on your time series, but they are not intended to cover all points.