search

py-why / dowhy

DoWhy is a Python library for causal inference that supports explicit modeling and testing of causal assumptions. DoWhy is based on a unified language for causal inference, combining causal graphical models and potential outcomes frameworks.
https://www.pywhy.org/dowhy
MIT License
7.06k stars 927 forks source link

Synth-Validation for Estimator Selection #63

Closed MaximilianFranz closed 5 years ago

MaximilianFranz commented 5 years ago

Hey there!

Have you considered / read the paper on Synth-Validation? I think including some similar method would benefit the DoWhy package as it helps to choose the right estimator given a specific dataset (especially, when no graph is given). We are working on an R implementation of the paper, which could possibly be included via RPy if you consider that something worthwhile at some point.

Either way, thanks for the awesome project!

amit-sharma commented 5 years ago

Thanks for your comment @MaximilianFranz ! Yes, synth-validation is an exciting idea. We are also exploring other methods for doing validation, such as bayesian model criticism and methods that introduce an interpretable parameter for confounding and then rerun the estimates.

Curious to hear your thoughts on these as we prioritize next steps for DoWhy.

And thanks for building out an R implementation of Synth-validation. Would love to integrate it with DoWhy. I am not an expert at RPy---Would you mind adding a Jupyter notebook that shows how to use the R implementation in python?

amit-sharma commented 5 years ago

@MaximilianFranz Wanted to share that we just released a roadmap for dowhy's development. Of course, synth-validation and related refutation methods are a high priority. Are you still interested in contributing to the repo?

Would also appreciate your feedback on the roadmap here: https://github.com/microsoft/dowhy/wiki/Roadmap

MaximilianFranz commented 5 years ago

Thanks for letting me know. I haven't forgotten about Synth-Validation, but the colleague who translated the code into R isn't sure about publishing it to the DoWhy repository yet. I will check again with him and our supervisor for further information. If I am not mistaken, the result of their evaluation was that the Synth-Validation method is not as effective as proposed, once the setting becomes more realistic. I'll keep you posted!

amit-sharma commented 5 years ago

Ah, alright! Thanks for the update @MaximilianFranz. Curious to hear more about the evaluation, that sounds really interesting. Would you be comfortable sharing the results?

MaximilianFranz commented 4 years ago

Late reply, sorry for that. My last information was that the institute was not comfortable publishing the R version of SynthValidation, as it might not be identical and is not robust. I am currently busy with implementing a framework for academic method evaluation using parametric DGPs (JustCause), so I won't be able to work on Synth-Validation anytime soon, though it would be cool to have!

amit-sharma commented 4 years ago

Ah, alright. Thanks for the update @MaximilianFranz. Wow, JustCause looks cool---evaluation of causal models is super important! If you think that there are ways DoWhy and JustCause can work together, let me know!