Add example to connect with CausalPy geo-lift?

[thipokKub]

Lift test is important for MMM model calibration, but it is I'm not sure on how to update prior to the experiment lift test. Recently pymc just released CausalPy, and I think it is a good idea to have an example on how to predict, evaluate, and iterate on the results

[twiecki]

Yes, we do plan to add this. Is this something you are interested in contributing?

[thipokKub]

I'm not sure, I don't know how exactly on a couple things

• How to add MMM model to CausalPy, but likely to do with this script template
• Not sure on how to iterate model after incremental test
  • In Lightweight_mmm, they didn't specify how to update their model after incremental testing. This can be seen missing from their flowchart (below)
  • In Robyn
    • in Model Calibration section, they have optimize using NRMSE , and decomp.RSSD
    • robyn_refresh() which will create new model consistent with previous data, and new data
    • Robyn has the option to extract trend + seasonality from the whole data, (using prophet) while only modeled using small window of media data (e.g. 2 years of data collection, but using only 3 months of media spending, see Model Window section)
  • In PyMC, the assumption on lift test is you know exactly what the conversion number on both control, and treatment group. such that the additional observation can be used to tuned model parameters. This can be done by adding additional likelihood function (with additional observation). But this can't be used with the original MMM model, as it will now need multiple outputs e.g.

In Robyn, I think it is because they use optimization based approach, they can customize, and update with the new data. But for Bayesian in PyMC, other than restart the whole model, I don't see how to update model to be consistent with the incremental testing results

[drbenvincent]

Hi @thipokKub. Would you be able to just expand a bit on your thoughts here? The issue title seems to focus on CausalPy and geolift (there is an example on that in the causalpy docs https://causalpy.readthedocs.io/en/latest/notebooks/geolift1.html), but the main content of the issue seems to focus more on how to incorporate lift test observations into the MMM.

[thipokKub]

Sorry, I've got a bit off-track. What I was trying to do is finding a repeatable testing, and calibration pipeline for MMM model, which is not exactly what I asked. But from what I've found from this pdf. The model need to be able to perform lift test, then recalibrated from the observed ROAS

I'm thinking that CausalPy can be used in for either synthetic control or difference-in-difference analysis specifically on ROAS/mROAS

———

Update

I think the best way to allow model training from lift experiment is to allow for multiple observations per time steps. I’ve tested this idea with the following gist

https://gist.github.com/thipokKub/54396626fdb1712573ffd4913ac2cd26

[drbenvincent]

I believe the lift test/calibration part of this is best suited for an issue in pymc-marketing.

We have an aspiration that CausalPy and PyMC-Marketing will synergise, and this sounds like it could be a particular example of that. Right now, we are not quite there, but I'm hoping to spend some time soon to ensure that we've got the right abstractions in CausalPy to be used by PyMC-Marketing.

Tagging @cluhmann, @lucianopaz, @ricardoV94, @juanitorduz for their thoughts.