pzivich
commented
5 years ago Generally, these plots are for diagnosing gross positivity violations. So if you look at a density plot like here, the overlap seems generally acceptable.
For an extreme violation, you can imagine the same plot as above but instead there is a bump in the blue plot around 0.7. Essentially, what this would tell me is that there is something fundamentally different about a subgroup of the treated versus the untreated. Once you identify that, you can dive into the issue. For example, maybe that subgroup was all age >50, but there were no individuals in the untreated group above age < 45. The plot is particularly useful for diagnostics in high dimensional models, since the subgroup maybe something more complicated (e.g., males, age > 50, and white were always treated).
I haven't thought too deeply about propensity vs logit plots, so these are my first thoughts. Your examples provided raise an issue more for propensity score matching (PSM), not the diagnostic necessarily. So the diagnostics purpose is to identify these positivity violations within our data (like above). Not necessarily distinguishing between 0.998 and 0.999. There is another diagnostic for this issue (it looks at the weights, rather that the probabilities).
As I currently see it:
positivity violations - IPTW.plot_kde() IPTW.plot_boxplot()
weight behavior - IPTW.positivity()
...although maybe these labels are not ideal since the positivity() is misleading. I chose that name based on Cole Hernan 2008.
However, your example does demonstrate one of the reasons to use IPTW over PSM. PSM may match 0.500 to 0.501 and 0.998 to 0.999, despite there being substantial differences between the probablities. IPTW allows these substantial differences ot occur (much like the coin flip examples you provided).
Much of my understanding comes from Cole Hernan 2008 but they don't specifically address this issue. I will ask around
I'd actually like to get your expert opinion on this question and answer: https://stats.stackexchange.com/questions/378876/why-is-it-easier-and-just-as-valid-to-assess-overlap-with-logit-propensities/
With that in mind, should we allow users to set if they want raw probabilities or log-odds?