Measure of optimal policy "stability"

[hhsievertsen]

Hi team,

thank you for developing this tool. It is super useful.

I noticed in my setting that the optimal policy changes slightly as I change the fraction of the sample I use to find the optimal policy and if change my see. It is of course not surprising that things change if we have different data points, but is there any way of assessing how "stable" an optimal policy is?

I didn't see any discussion of this in Athey and Wager's Econometrica paper, but I might have missed it. Currently I am thinking of drawing X samples of size N of my data and compare the optimal policy across these X samples, but that seems a bit ad hoc (and it difficult to compare the optimal policies across samples).

Thanks again for developing this! Hans

[erikcs]

Hi @hhsievertsen, if you look at Figure 2 in https://arxiv.org/pdf/1702.02896v5.pdf (earlier draft of that paper) @susanathey and @swager do a simple stability exercise there.

[hhsievertsen]

Thanks @erikcs!