DeepAR external Regressors

[cna274]

I am working on time series forecasting using LSTMs. My dataset has time series (sales on each day) data, plus external regressors like discount on particular day, holidays, day of week. Now that I want to move to DeepAR, I was wondering, how to incorporate these features in the DeepAR training data (json format) and run DeepAR ? "cat": in training dataset didn't work. Note : I want to add these because, I want to know how forecast is affected based on discount and holiday period features.

[djarpin]

Thanks, @cna274 . Currently, the SageMaker DeepAR algorithm does not support external regressors. However, some lags are automatically calculated and included in the model for you. With daily data, the lags included are previous week, 2 weeks, 3 weeks, 4 weeks, and year. Ideally, this should capture some of the seasonality. Unfortunately, you won't be able to capture the effect of discount, or planned discount, in your DeepAR model.

[pjebs]

https://aws.amazon.com/blogs/machine-learning/amazon-sagemaker-deepar-now-supports-missing-values-categorical-and-time-series-features-and-generalized-frequencies/

This blog was posted on 11 JUL 2018. Has this situation now changed @djarpin ?

[pjebs]

The blog post talks about: "Custom time series feature" which seems like an external regressor to me

[djarpin]

Hi @pjebs - Correct. DeepAR began supporting external regressors in July.

[pjebs]

Is it possible to obtain the model directly without accessing it through the endpoint?

[djarpin]

Hi @pjebs - The model.tar.gz is written to S3 output path you specify. What would you like to do with the model though? DeepAR is a recurrent neural network model, so there aren't really interpretable coefficients like you'd find in a linear regression. And the scoring code to generate predictions and quantiles is non-trivial. So, it would be difficult to generate predictions locally. There is SageMaker batch transform which would allow you to generate predictions without having a persistent endpoint. Thanks.

[pjebs]

Our use case is we have sales data for lots of different customers and would like to forecast for each individual customer. It just seems cumbersome for our backend (or frontend) to have to keep a record of each generated forecasting endpoint when we want to apply it for each customer.

[djarpin]

Hi @pjebs - Typically DeepAR excels when building one model to forecast multiple time series. i.e. You may want to build one DeepAR model capable of generating forecasts for every customer. This would mean you'd only have on endpoint to keep a record of. Is there a reason you want to have a separate model for each customer? Thanks.

[pjebs]

In general, wouldn't each customer have totally different sales characteristics (different countries, different locations, different markets, different industries etc). Are you saying for DeepAR, just stick each customer's data into the one input as different columns?

Given each customer could have different timezones, would we need to convert the dates column to utc and have a different category for timezone?

[djarpin]

Hi @pjebs - To a large extent this is dependent on your data. In some cases, even if customers are different, overarching similarities like increased buying on weekends or holidays, or upward or downward trends in aggregate demand over time, means that fitting one model in aggregate will produce better results. A bit of experimentation here, in particular using DeepAR's "cat" feature to group customers from the same market and/or industry, may help you determine what works best for your data.

[pjebs]

@basharre

[cna274]

@djarpin Yes you are right, we can also add Multiple categories per time series now in DeepAR. For example in your case you can group customer from one country to particular category.

Given each customer could have different timezones, would we need to convert the dates column to utc and have a different category for timezone? This is very specific design question. But idea for DeepAR is cross-learning hence high number of time series would give better accuracy.

I added external regressors and ran the model, works perfectly fine. I am figuring out hyper-parameter tuning now. I see "start_date" in input json is very important make sure you give right value to it.