search

py-why / dowhy

DoWhy is a Python library for causal inference that supports explicit modeling and testing of causal assumptions. DoWhy is based on a unified language for causal inference, combining causal graphical models and potential outcomes frameworks.
https://www.pywhy.org/dowhy
MIT License
6.99k stars 923 forks source link

Expose interventional outcomes from do operator for further analysis #1011

Closed drawlinson closed 1 year ago

drawlinson commented 1 year ago

My users on causal wizard (which uses DoWhy) have often requested counterfactual outcomes such as: "what's the total cost if none or all subjects were treated?". These can be answers by DoWhy using the Do-operator by adjusting the treatment value and passing in various subsets of the data. However, the response from Do-operator is not sufficiently detailed - it's the new average effect size.

The average (mean) effect size is not always the metric of interest. For example, the sum of outcomes is more often useful. Consider the case where the various counterfactual scenarios represent different maintenance investment policies over many infrastructure assets. In this case, we want to know the total maintenance cost (i.e. the sum of all cost outcomes given intervention).

In many cases, people want to know all the predicted outcomes individually. For example, if you wanted to perform validation with held-out data, you could compare the predicted results to the actual results.

The implementation of the do-operator in compatible estimators (those descended from RegressionEstimator) first computes all interventional outcomes, and then calculates the mean and returns it. So the desired results are all there - they're just inaccessible.

One possible solution would be to have an interface which returns the list of interventional outcomes e.g. 

class RegressionEstimator
  ...

  def get_interventional_outcomes(self, treatment_val, data_df):
    ...

  def _do(self, treatment_val, data_df):
    interventional_outcomes = self.get_interventional_outcomes(treatment_val, data_df)
    return interventional_outcomes.mean()

This is probably ideal for also allowing analysis of individual outcome predictions given interventions.

However, another possibility is to simply modify the do interface to return mean, sum and count as these are the most interesting quantities and satisfy most counterfactual use-cases:

class RegressionEstimator
  ...

  def _do(self, treatment_val, data_df):
    ...

    interventional_outcomes = self.predict_fn(self.model, new_features)
    return interventional_outcomes.mean(), interventional_outcomes.sum(), len(interventional_outcomes)

Happy to create a PR for either method, or another suggestion, if it's wanted.

It would also be nice to have support for the do-operator in other estimators, but I think that's a separate issue.

amit-sharma commented 1 year ago

Hey @drawlinson that's a great point. Thanks for raising it. Users can benefit from having access to the individual outcomes.

Among the options, I like the first one since it does not change the output signature of do method. For simplicity in naming, I suggest calling the new method interventional_outcomes (remove get). Look forward to your PR on this.

drawlinson commented 1 year ago

@amit-sharma PR at https://github.com/py-why/dowhy/pull/1018

I found test coverage for the change via RegressionEstimator.estimate_effect(), which is used in several tests such as tests/test_notebooks.py::test_notebook[dowhy_estimation_methods.ipynb]

I couldn't find a way to link this issue to the PR. Do let me know if I can do anything else. Thanks!