Absolute effect - Average positive and cumulative negative

[FlavioZanette]

Hi, @WillianFuks, first I would like to congratulate you for the great work done in this lib, I am using it frequently to evaluate results of experiments.

Currently, I'm trying to evaluate results after starting to use a system, the evaluation period is long. I have only one control group and I notice a big oscillation between them.

I have a doubt about the results of the summary, I have had positive average results and negative cumulative ones, I analyzed how the lib was coded and the results are not making sense. Can you help me with this? is there any explanation?

I use the fit_method as hmc, but without it the result are almost the same.

I also tried an approach using the forecast of the series itself, generated in the prophet, as a control group. The results are similar.

Best, Flávio

[WillianFuks]

Hi @FlavioZanette ,

The results you are observing are weird indeed, I'm not entirely sure what happened. One thing that catches the eyes is that the cumulative standard deviations are huge and its plotting graphic seems broken, not sure what could have caused it.

Just wondering, is it possible for you to apply some masking (such as a white noise) in your data and sent if over to me? It could be over email as well so it's safer (I'm willian.fuks at gmail). That way I'd be able to trace each step of the execution to see if I find something.

Best,

Will

[WillianFuks]

Hi @FlavioZanette ,

Thanks for sending the mock data to me. What happened here is that your data ended up hitting a branch case not yet covered by the package, which is data containing "holes" in the post intervention data.

For instance, your data had values for '2021-09-03', '2021-09-04' and '2021-09-06' but day '2021-09-05' was not present.

This made tfcausalimpact process everything as expected but it added NaN values in those holes. When computing cumulative data, it eventually finds those null values and nullifies everything after it sees the first NaN.

That's why you observed the weird results where absolute effect was positive (it just takes an average where null points does not affect anything) and negative cumulative effects that were affected by those same null values.

Now version 0.0.11 is supposed to proactively remove those points from the analyzes, which should stabilize everything (install it with pip install -U tfcausalimpact).

You also asked me about the wide range of confidence intervals in the inferences being made, this can be the result of a few things. As this is quite empirical, you'd need to try out several things to see if some actions help. Things to consider:

• It could be that the linear regressors (covariates) doesn't explain the data to an extent that the random walk component stabilizes over time. You could try arranging more covariates to see if it helps.
• There are differences between the vi and hmc methods, it's recommended to try both.
• Default prior for local level is 0.01. You could try making it a bit smaller, such as 0.001. This is only recommended if the covariates are expected to explain well your data, otherwise you are just forcing the uncertainty in your model to be inadvertently removed.
• Other models components might help better explain data, such as local linear levels and seasonal components.

You could play with those ideas to see if any helps.

As a final note, I noticed you use "test" and "control" to identify your response and covariate. Just as a reminder, strictly speaking those names are not entirely suited there because in this causal analyses package the control is past observed data and test is post intervention. The correct concepts would be something like ["total sales", "budget invested in marketing"] for instance. The first is what is observed and investigated and the rest are variables that try to explain the first. I'm just touching briefly on that to make sure you are not using wrong data in your analyses.

I hope that helps, please let me know if everything is working as expected now.

Best,

Will

[FlavioZanette]

Hi @WillianFuks thank you so much for the quick answers and to have fixed the lib in this new version. It's working perfectly, I'll work on the comments you made to try to reduce the p value of my test.