search

py-why / dowhy

DoWhy is a Python library for causal inference that supports explicit modeling and testing of causal assumptions. DoWhy is based on a unified language for causal inference, combining causal graphical models and potential outcomes frameworks.
https://www.pywhy.org/dowhy
MIT License
6.88k stars 916 forks source link

Inconsistency in the placebo_treatment_refuter when using estimate_effect of IV #1180

Open xwbxxx opened 1 month ago

xwbxxx commented 1 month ago

I tried to reproduce the experimental results of this example A Simple Example on Creating a Custom Refutation Using User-Defined Outcome Functions and replace it with a different refuter, i.e.placebo_treatment_refuter. Here is the graph structure. image Since I specified the parameter method_params={'iv_instrument_name': 'Z0'} in causal_estimate_iv_1, placebo_treatment_1 derived a very strange result of new effect value, which should have been close to zero, but turned out to be in the thousands and quite different from the result without specifying the parameter method_params (i.e. causal_estimate_iv_2 and placebo_treatment_2 ).

causal_estimate_iv_1 = model.estimate_effect(identified_estimand,
                                             method_name="iv.instrumental_variable",
                                             method_params={'iv_instrument_name': 'Z0'})

placebo_treatment_1 = model.refute_estimate(identified_estimand, causal_estimate_iv_1,
                                            placebo_type="permute",
                                            method_name="placebo_treatment_refuter")
causal_estimate_iv_2 = model.estimate_effect(identified_estimand,
                                             method_name="iv.instrumental_variable", )
placebo_treatment_2 = model.refute_estimate(identified_estimand, causal_estimate_iv_2,
                                            placebo_type="permute",
                                            method_name="placebo_treatment_refuter")

According to the document, the parameter method_params={'iv_instrument_name': 'Z0'} can be omitted in this scenario where there is only one instrumental variable Z0. So the inconsistency of the results must be due to the existence of a bug. image

I spent a long time debugging to trace back the cause of the inconsistency. Eventually, I realized that the problem lied in the code at line 97-99 of \dowhy\causal_estimators\instrumental_variable_estimator.py. image In line 97, when executing placebo_treatment_refuter given instrumental_variable_estimator, self.estimating_instrument_names will be assigned ['placebo_Z0']. However, in line 98-99 the existance of self.iv_instrument_name will overwrite the assignment ['placebo_Z0'] with ['Z0'] and result in the huge mistake in the following step of effect estimate, because the placebo treatment and placebo iv should maintain consistent when placebo_type="permute"

amit-sharma commented 1 month ago

Thanks for the thorough debugging of this issue, @xwbxxx . This is indeed a bug. Will try to fix this in a future PR