Add shock to the model / parametric extra regressors

[zippeurfou]

Hey, This is more of a feature suggestion than actually an issue. I was wondering if we could simulate shock to the system. Right now the doc suggest to just remove them. However sometimes you might not want to do that. An example could be trying to forecast Youtube traffic where you can have multiple shock coming up in small period of time (which would correspond on new video being published for example). A way to solve it would be to add the ability to have "shocks" to your model and for example fit them with exponential decay (which could describe correctly a new post being viral, a new video posted on youtube and so on). i.e. something like median * delta * exp(-( i - t0[j] ) / tau) where t0 is a vector that represent all the "initial" shock time. Removing the trend, seasonality and holiday effect it would look like this: It might be a bit out of the scope of prophet but I thought I'd ask anyways.

[ryantsullivan]

This would be really interesting for the data I work with which is admissions data for movie theaters. We see shocks, maybe not as extreme as a YouTube video, when an Avengers or Star Wars (except solo) come out.

[bletham]

This is a really interesting example. I think with some effort something along these lines could be done, albeit not the entirely ideal thing.

If I had a completely known form for the shock, then I could just subtract it out and proceed with forecast as usual. In this case, let's say it's a*exp(-kt) at each of the known shock points, and I'm saying that I know both a and k at each of the (also known) shock times.

The first relaxation would be to assume k known, but not the magnitude of the shock a. This situation can be handled already with Prophet by specifying an extra regressor for each of the shocks, each of which would contain exp(-kt) starting at that shock date. Prophet will then learn the coefficient a, and when you generate the components plot the extra_regressors panel will look exactly like the figure above.

Ideally we would want to fit both a and k. There isn't yet a way to learn parametric extra regressors. I think that'd be great, but since it would have to go in the Stan I suspect it will be challenging. Something that might work, however, would be this:

Choose a set of K values for k that you expect each shock would be well enough described by one of those values (not too many, maybe K=5ish?). Now, create an extra regressor for each shock and k combination. So with S shocks, you'll be adding S*K extra regressors, which will be exp(-kt) with that value of k and starting at that shock time. Now, your actual model for the effect at shock s will be

a_1 exp(-k_1 * t) + ... + a_K exp(-k_K * t)

and Prophet would be fitting the as. With L1 regularization this might give you something very close to actually fitting a and k in the original model. Prophet will apply L2 regularization so you'll end up with the shock being a combination of these different decay rates, but it could potentially still fit well.

If anyone has data with this property and is able to try it out I'd love to see how it works. Let me know if anything is unclear.

[zippeurfou]

Don't have the time right now but I can share the data generation process for this chart:

generate_data <- function(n_point=100,noise_mu=1000,noise_sigma=5,n_peak=5,coef_shape=5,coef_rate=1,tau_min=3,tau_max=10){
  idx <- rbinom(n_peak,n_point, runif(n_peak,0,1))
  noise <-  rnorm(n_point,noise_mu,noise_sigma)
  init_data <- noise 
  coef_delta <- rgamma(n_peak,coef_shape,coef_rate)
  tau <- runif(n_peak,tau_min,tau_max)
  data <- init_data
  for (i in seq_along(idx)){
    v <- idx[i]
    delt <- coef_delta[i]
    t <- tau[i]
    data <- data + median(init_data) * delt * as.numeric(seq(1,n_point) >= v ) * c( rep(0,v-1),   exp(-c(seq(0,n_point-v)/t    )))   
  }
  list(
    data = data,
    tau = tau,
    delta = coef_delta,
    peak_idx = idx,
    noise = noise
  )
}
generated <- generate_data()

data <- generated$data

plot(data,t="l")

[JasonAizkalns]

I am currently working through this exact use-case / model -- the context being pricing shocks (increases and decreases) and their lagged and/or decaying effect on demand modeling. I independently arrived at the same approach as @bletham with the "art" being choosing an appropriate k. While I'm not at the liberty to share actual data, please let me know if I can help in any way.

[zippeurfou]

@JasonAizkalns if you came up with a stan model it could be beneficial to share it I believe.

[trilliumtechnical]

@bletham You mentioned above that (June 8 2018) that you would be interested in some data with a particular pattern of shocks. I have data like that if you are still interested.

[bletham]

@trilliumtechnical Like I described above, parametric extra regressors will require some pretty significant changes and so it isn't very high on the to-do list, but having a data example would certainly be helpful to be able to work towards once the other pieces are in place!