search

WillianFuks / tfcausalimpact

Python Causal Impact Implementation Based on Google's R Package. Built using TensorFlow Probability.
Apache License 2.0
591 stars 70 forks source link

output of HMC and VI, #95

Open aazz7777 opened 2 months ago

aazz7777 commented 2 months ago

Hello, Will,

When I run code using VI, and HMC, it show totally opposite casual impact VI - for most of sample, and HMC is +, Would anyone can help me to understand why?

WillianFuks commented 2 months ago

Hi @aazz7777 ,

This is expected indeed. The VI algorithm is mainly used for quick experimentation. Once you have confirmed that the model is working then it's recommended to use the HMC model which is slower but precise (and stable).

aazz7777 commented 2 months ago

Thank you for the answer. Also, when I use the HMC , the impact turns out to be totally opposite compare to the case when I run the same data in R casual impact package. For example, if R said (-) impact, HMC returns (+) for many ample cases. I was expecting similar output since HMC is a special variant of MCMC. I'm wondering how I interpreted or handled it. because we can't say one is more accurate. and also, the probability is really difference. Python said not significant, and R said significant... I'm running many test cases.

WillianFuks commented 2 months ago

This is unexpected indeed. Is it possible to share a transformed dataset for experimenting on my side? Both packages should converge to same conclusions.

aazz7777 commented 2 months ago

I don't have authority to the data so I'm not sure I can share the dataset... but I can show the output of probability and impact of with R, VI, and HMC.

aazz7777 commented 2 months ago

image

This is sample output in R casual impact and your VI and HMC result. It can lead pretty different results when it comes to business.. some test results are similar.and many are different when I test I set both 0.05 I saw your package can't change alpha. since it only affect CI.

WillianFuks commented 2 months ago

Hi @aazz7777 ,

Results from VI are indeed less stable. The ones that should be considered are the ones from HMC

aazz7777 commented 2 months ago

Thank you for the commend, another my question is that case 1: R -8 (negative impact) 4.72 is positive impact Case2: R : negative impact -9, HMC positive impact 2.26 but one said significant, another said not significant. Case 3, both negative impact, R said significant, another said HMC said not significant. if this is the case, which one need to be chosen as recommendation ?? for example, both said negative impact, but one said significant, another said not significant?