Open thipokKub opened 1 year ago
Yes, we do plan to add this. Is this something you are interested in contributing?
I'm not sure, I don't know how exactly on a couple things
NRMSE
, and decomp.RSSD
robyn_refresh()
which will create new model consistent with previous data, and new data2
years of data collection, but using only 3
months of media spending, see Model Window section)likelihood
function (with additional observation). But this can't be used with the original MMM model, as it will now need multiple outputs e.g.
In Robyn, I think it is because they use optimization based approach, they can customize, and update with the new data. But for Bayesian in PyMC, other than restart the whole model, I don't see how to update model to be consistent with the incremental testing results
Hi @thipokKub. Would you be able to just expand a bit on your thoughts here? The issue title seems to focus on CausalPy and geolift (there is an example on that in the causalpy docs https://causalpy.readthedocs.io/en/latest/notebooks/geolift1.html), but the main content of the issue seems to focus more on how to incorporate lift test observations into the MMM.
Sorry, I've got a bit off-track. What I was trying to do is finding a repeatable testing, and calibration pipeline for MMM model, which is not exactly what I asked. But from what I've found from this pdf. The model need to be able to perform lift test, then recalibrated from the observed ROAS
I'm thinking that CausalPy can be used in for either synthetic control or difference-in-difference analysis specifically on ROAS/mROAS
———
I think the best way to allow model training from lift experiment is to allow for multiple observations per time steps. I’ve tested this idea with the following gist
https://gist.github.com/thipokKub/54396626fdb1712573ffd4913ac2cd26
I believe the lift test/calibration part of this is best suited for an issue in pymc-marketing.
We have an aspiration that CausalPy and PyMC-Marketing will synergise, and this sounds like it could be a particular example of that. Right now, we are not quite there, but I'm hoping to spend some time soon to ensure that we've got the right abstractions in CausalPy to be used by PyMC-Marketing.
Tagging @cluhmann, @lucianopaz, @ricardoV94, @juanitorduz for their thoughts.
Lift test is important for MMM model calibration, but it is I'm not sure on how to update prior to the experiment lift test. Recently pymc just released CausalPy, and I think it is a good idea to have an example on how to predict, evaluate, and iterate on the results