Prior SD Determination

[BrianMiner]

I was reading your original post and it looks like you must be running the causualimpact package over a grid of prior SD and plotting MAPE, DW. Is this done just for the pre-period? If so, do you think it better to try and determine the best level out of sample (Cross validation)? Do you find that it is useful to change the SD based on these findings?

[klarsen1]

I’m not sure it makes sense to treat the post period as a holdout sample. The point of the package is to find the markets that align most closely in the pre-period and the measure the lift from the intervention in the post-period. I’m not crazy about the idea of tuning the model by peeking into the model fit of the future (the period in which we’re supposed to monitor lift).

Also, the DW — a measure of autocorrelation at lag 1 — inherently makes more sense for the training period as we’re merely trying to test if the model is able to produce white noise errors. If the DW is far from 2, I’d have less confidence in the post period learnings.

Kim

On Wed, Oct 24, 2018 at 12:32 PM BrianMiner notifications@github.com wrote:

I was reading your original post and it looks like you must be running the causualimpact package over a grid of prior SD and plotting MAPE, DW. Is this done just for the pre-period? If so, do you think it better to try and determine the best level out of sample (Cross validation)? Do you find that it is useful to change the SD based on these findings?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/klarsen1/MarketMatching/issues/9, or mute the thread https://github.com/notifications/unsubscribe-auth/AJqqdJb1WEWlZAJkGIeH7S_B8bjpoh-Pks5uoMBbgaJpZM4X4pkI .

[BrianMiner]

Yeah - I was not thinking of prediction of the post period. More thinking of cross validating the pre-period to determine the settings.

Also - in the cases I have seen, DW tends to be around 1.2 - 1.3. Would you distrust?

On 10/29/2018 12:34 PM, Kim Larsen wrote: I’m not sure it makes sense to treat the post period as a holdout sample. The point of the package is to find the markets that align most closely in the pre-period and the measure the lift from the intervention in the post-period. I’m not crazy about the idea of tuning the model by peeking into the model fit of the future (the period in which we’re supposed to monitor lift).

Also, the DW — a measure of autocorrelation at lag 1 — inherently makes more sense for the training period as we’re merely trying to test if the model is able to produce white noise errors. If the DW is far from 2, I’d have less confidence in the post period learnings.

Kim

On Wed, Oct 24, 2018 at 12:32 PM BrianMiner notifications@github.commailto:notifications@github.com wrote:

I was reading your original post and it looks like you must be running the causualimpact package over a grid of prior SD and plotting MAPE, DW. Is this done just for the pre-period? If so, do you think it better to try and determine the best level out of sample (Cross validation)? Do you find that it is useful to change the SD based on these findings?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/klarsen1/MarketMatching/issues/9https://github.com/klarsen1/MarketMatching/issues/9, or mute the thread https://github.com/notifications/unsubscribe-auth/AJqqdJb1WEWlZAJkGIeH7S_B8bjpoh-Pks5uoMBbgaJpZM4X4pkIhttps://github.com/notifications/unsubscribe-auth/AJqqdJb1WEWlZAJkGIeH7S_B8bjpoh-Pks5uoMBbgaJpZM4X4pkI .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/klarsen1/MarketMatching/issues/9#issuecomment-433979502, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AIZkgibcCfcYVXfgCkaAYnY2iT3J-HyBks5upy4QgaJpZM4X4pkI.

[klarsen1]

Would like to see a number closer to 2. Not sure I would distrust, but I’d be concerned.

Try lowering the prior SD of the level term.

On Mon, Oct 29, 2018 at 5:55 PM BrianMiner notifications@github.com wrote:

Yeah - I was not thinking of prediction of the post period. More thinking of cross validating the pre-period to determine the settings.

Also - in the cases I have seen, DW tends to be around 1.2 - 1.3. Would you distrust?

On 10/29/2018 12:34 PM, Kim Larsen wrote: I’m not sure it makes sense to treat the post period as a holdout sample. The point of the package is to find the markets that align most closely in the pre-period and the measure the lift from the intervention in the post-period. I’m not crazy about the idea of tuning the model by peeking into the model fit of the future (the period in which we’re supposed to monitor lift).

Also, the DW — a measure of autocorrelation at lag 1 — inherently makes more sense for the training period as we’re merely trying to test if the model is able to produce white noise errors. If the DW is far from 2, I’d have less confidence in the post period learnings.

Kim

On Wed, Oct 24, 2018 at 12:32 PM BrianMiner <notifications@github.com

mailto:notifications@github.com wrote:

I was reading your original post and it looks like you must be running the causualimpact package over a grid of prior SD and plotting MAPE, DW. Is this done just for the pre-period? If so, do you think it better to try and determine the best level out of sample (Cross validation)? Do you find that it is useful to change the SD based on these findings?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/klarsen1/MarketMatching/issues/9< https://github.com/klarsen1/MarketMatching/issues/9>, or mute the thread < https://github.com/notifications/unsubscribe-auth/AJqqdJb1WEWlZAJkGIeH7S_B8bjpoh-Pks5uoMBbgaJpZM4X4pkI < https://github.com/notifications/unsubscribe-auth/AJqqdJb1WEWlZAJkGIeH7S_B8bjpoh-Pks5uoMBbgaJpZM4X4pkI>

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub< https://github.com/klarsen1/MarketMatching/issues/9#issuecomment-433979502>, or mute the thread< https://github.com/notifications/unsubscribe-auth/AIZkgibcCfcYVXfgCkaAYnY2iT3J-HyBks5upy4QgaJpZM4X4pkI>.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/klarsen1/MarketMatching/issues/9#issuecomment-434133488, or mute the thread https://github.com/notifications/unsubscribe-auth/AJqqdHBFqKM-lKawE-gMBbXTvVO7he2Uks5up6NlgaJpZM4X4pkI .