pymc-labs / CausalPy

A Python package for causal inference in quasi-experimental settings
https://causalpy.readthedocs.io
Apache License 2.0
829 stars 52 forks source link

Add a temporal validation period to synthetic control and interrupted time series experiments #367

Open drbenvincent opened 1 week ago

drbenvincent commented 1 week ago

TODO


๐Ÿ“š Documentation preview ๐Ÿ“š: https://causalpy--367.org.readthedocs.build/en/367/

review-notebook-app[bot] commented 1 week ago

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

drbenvincent commented 1 week ago

Good progress so far. But we are missing separate Bayesian $R^2$ for training and validation phases.

Synthetic control figure:

Screenshot 2024-06-21 at 14 51 07

Interrupted time series figure:

Screenshot 2024-06-21 at 14 50 53
codecov[bot] commented 1 week ago

Codecov Report

Attention: Patch coverage is 90.47619% with 4 lines in your changes missing coverage. Please review.

Project coverage is 85.81%. Comparing base (f6fd97c) to head (444e363).

Files Patch % Lines
causalpy/pymc_experiments.py 83.33% 4 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #367 +/- ## ========================================== + Coverage 85.60% 85.81% +0.20% ========================================== Files 22 22 Lines 1716 1748 +32 ========================================== + Hits 1469 1500 +31 - Misses 247 248 +1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

drbenvincent commented 1 week ago

Based on one of @cetagostini 's PR's (https://github.com/pymc-labs/CausalPy/pull/368), I'm wondering if we should add a small feature to calculate a ROPE based on the validation period. Something a bit like this:

Screenshot 2024-06-24 at 14 36 21

Any thoughts/comments welcome. I'm not convinced this is a good idea yet - especially because once we add in actual time series models then the credible interval will increase as we forecast further into the future.