WillianFuks
commented
2 years ago Hi @DavidJ005 ,
A p-value lower than 5% is an indication that observing the post-intervention period by random chance is unlikely. This means that whenever the result is lower than 5% the conclusion is that there was a statistical valid impact (for instance, a marketing campaign did increase sales as observing the same post-intervention points without the marketing campaign would be unlikely).
A strong correlation between X and y may be helpful for extrapolating what would be observed in the post-intervention period without the impact. So if you have (supposedly) good covariates then the results may be more reliable. If you found that the likelihood of observing the post-intervention data by random chance is 32% this is considered too high and therefore there isn't much evidence in your case that the intervention had a real impact after all (you would probably sell the same quantities with of without the marketing campaign, for instance).
As the report message states, you could also play around with the amount of days for the post-intervention period to investigate the balance between noise and signal to see if results change; notice that the conclusion tends to be an empirical evaluation of your data and interpretation of the results.
Hope that helps,
Best,
Will
DavidJ005
Hi @WillianFuks and thank you for that package,
I was wondering if tfcausalimpact is capable of saying "the event has no impact on Y" with a p-value<0.05 ? For example: I am running tfcausalimpact between a Y and X that has a 0.93 pearson correlation for the pre-period. So I am confident about the quality of X to be link to Y and the p-value is 0.32.
Knowing that, can I say "no effect" or I can just say "I don't know". I know it's not an question related to the package, but I am kind of stuck with that question, if you could help it will be great.
ps: I put the full report below (I can't share the data).
Analysis report {CausalImpact}------------------------------------------------------
During the post-intervention period, the response variable had an average value of approx. 327184.03. In the absence of an intervention, we would have expected an average response of 308917.78. The 95% interval of this counterfactual prediction is [210992.7, 394620.11]. Subtracting this prediction from the observed response yields an estimate of the causal effect the intervention had on the response variable. This effect is 18266.25 with a 95% interval of [-67436.08, 116191.33]. For a discussion of the significance of this effect, see below.
Summing up the individual data points during the post-intervention period (which can only sometimes be meaningfully interpreted), the response variable had an overall value of 981552.12. Had the intervention not taken place, we would have expected a sum of 926753.38. The 95% interval of this prediction is [632978.11, 1183860.33].
The above results are given in terms of absolute numbers. In relative terms, the response variable showed an increase of +5.91%. The 95% interval of this percentage is [-21.83%, 37.61%].
This means that, although the intervention appears to have caused a positive effect, this effect is not statistically significant when considering the entire post-intervention period as a whole. Individual days or shorter stretches within the intervention period may of course still have had a significant effect, as indicated whenever the lower limit of the impact time series (lower plot) was above zero.
The apparent effect could be the result of random fluctuations that are unrelated to the intervention. This is often the case when the intervention period is very long and includes much of the time when the effect has already worn off. It can also be the case when the intervention period is too short to distinguish the signal from the noise. Finally, failing to find a significant effect can happen when there are not enough control variables or when these variables do not correlate well with the response variable during the learning period.
The probability of obtaining this effect by chance is p = 32.47%. This means the effect may be spurious and would generally not be considered statistically significant.