04/07: Max Farrell

[ehuppert]

Comment below with a well-developed question or comment about the reading for this week's workshop. These are individual questions and comments.

Please post your question by Wednesday 11:59 PM, and upvote at least three of your peers' comments on Thursday prior to the workshop. You need to use 'thumbs-up' for your reactions to count towards 'top comments,' but you can use other emojis on top of the thumbs up.

[JadeBenson]

Thank you so much for your interesting research and presenting! I think combining causal inference methods with deep learning is quite exciting. At the end of your article you mention that this serves "as a first step in the exploration of deep learning as a tool for economic applications." What do you think are the most exciting potential applications and/or applications that you have since seen after this article's publication in academia/industry?

[javad-e]

Thank you for presenting your truly innovative projects at our workshop! My questions are about the role of deep learning in future econometrics. As researchers continue to demonstrate the potential applications of deep learning in economics, how do you think econometric models will be affected? Do you think some traditional econometric models will be replaced with deep learning? Do you think the literature on deep learning for causal inference is rigorous enough to be accepted as reliable by economists? And what are the drawbacks of using deep learning for causal inference?

[FranciscoRMendes]

Could you please explain how this is different from simply creating a special embedding layer and building the model on top of that? The only difference I can see is that some "links" between the input layer and the embedding layer would be severed, not only is that tough to do in an architecture but if you knew that, you would not really need this new representation.

[Hai1218]

Thank you for sharing your work! I must say that struggled with understanding the mathematical/substantive part of this paper but I appreciate quite a lot the implications to the change of paradigm of social sciences these neural networks and deep learning techniques could bring. I would love to hear more about some of the potential applications of the methodology you developed, similar to the Marketing example you gave in one of the papers. Relating to the field of political science and election forensics, could the deep leaning techniques be used to predict election results? Or to detect anomalies in election activities?

[sabinahartnett]

Thank you for sharing your work with us! I assume that much of the heterogeneity you reference in the article arises from humans behaving in divergent/inconsistent ways (making DNN so preferable over perhaps simpler ML models). These economic models seem to be largely built on prior behaviors, I'm wondering if you see value in incorporating more cognitive modeling / investigating the input into individual decisions to improve model accuracy / prediction on higher level tasks?

[PAHADRIANUS]

Thank you for providing your progress with us. In my naïve understanding of your work, it seems that your application of DNNs mainly tackles nonparametric, heterogenous inputs in microeconomic modeling. How can one extend your framework to possibly conduct a more empirical study of real data? Or even analyze macro status and revise financial policies？

[Qiuyu-Li]

Dear Prof. Farrell: To be honest, I didn't quite understand the paper, but I can see that it pictures a very promising future where deep learning may complement bad experiment designs in causal inferences. I'm very looking forward to your presentation and hope I can grab more sense of it. Thank you for coming to our workshop!

[erweinstein]

In Deep Neural Networks for Estimation and Inference, you demonstrate the applicability of your results using the same direct-mail marketing experiment data from Hitsch and Misra 2018. I agree that the challenge of attempting to estimate the treatment effect of customers receiving the catalog (vs. not; and/or vs. an alternative where only previous loyal customers receive it) is an excellent real-world example of the problem of causal inference. However, is it possible that the specifics of the direct-mail experiment fit your methods very well, but wider applicability is more limited? (It would still be true and quite significant that your preferred deep ReLU net approach outperforms the models studied in Hitsch and Misra 2018.) For example, you and your co-authors make a few passing references to health and medicine (as someone very interested in that application domain, I appreciate that). I could make the case that some types of causal inference questions in a medical setting might in fact be very similar to the direct-mail marketing experiment, say if a large hospital or medical group is considering an "A/B test" of a different set of protocols for how to treat otherwise-healthy patients at risk for colon cancer. In addition to the clear experimental nature of that example, the overall proportion of those patients we'd expect to die from colon cancer is similar to the proportion of customers who end up buying something from that company (>5% but <10%). But many other healthcare examples, like the variety of small, everyday diagnostic and treatment decisions studied by Mullainathan and Obermeyer or Currie and MacLeod seem far less similar, and thus less amenable. (E.g., would you have to wait for someone to run a large-scale experiment on different versions of each those type of decisions in order to apply your method?) Given that you, your co-authors, and your other Booth colleagues seem to pay more attention to possible health/medical applications than others, perhaps you already have some ideas or work-in-progress addressing this...

[william-wei-zhu]

Hi Professor Farrell, thank you very much for sharing your work. We'd like to hear your thoughts on the future research direction of improving the casual inference capabilities within deep learning. Looking forward to your presentation.

[qishenfu1]

Hi Prof. Farrell: As a MACSS-Econ student, I am very happy to see an econ-related speaker in our computational workshop. I have a general question -- machine learning methods are usually considered black boxes. What approaches do you think can be used to better interpret the machine learning algorithms?

[JoeHelbing]

I'll be honest, the papers were both quite above my level of understanding, but it's something I hope to be able to study more in depth in the future, which is sort of what brought this question to mind. It seems that within fields such as political science and others there's a certain reticence to use ML, even more so deep neural networks. Interested in how you think these tools can be adapted or explained to a point that researchers less exposed to this computational toolset could understand the methodology, and thus have confidence in the results. I see a potential issue in "breaking into" these fields simply because the methods are so outside the core field that the it not really reviewable by a sociology/poli sci/etc. journal.

[Lynx-jr]

Oops, just recalled I nominated Prof. Farrell last year because I'd really want MACSS folks to discuss if there exist good applications for his research. I knew his papers are very econometrics so hope people like them ;) My question would be that compared with Bertrand et al. 2010, what other obvious benefits does your inference framework provide?

[taizeyu]

Hi Dr Farrel. Thank you very much for sharing your work. It is such an excellent job. My question is that why do you think it is suitable for you to use "Monte Carlo analysis and an empirical application to direct mail marketing" to prove the causal inference model. In addition, what are applications can we use with the model you created.

[edelahayeUChicago]

Professor,

I enjoyed reading your papers and am looking forward to your presentation tomorrow! The part of your papers that caught my eye the most was your discussion of "Optimal Targeting" using your model to determine the best level of prior spending to target with catalogues so as to maximise profits.

You mention that this can be expanded to many other covariates to identify even more granular groups that are ripe for focus by the firm, however I was wondering if this could also be extended to other treatments (i.e. choosing treatment for a group, rather than group for a treatment). Given the large range of ways a treatment could be designed, are there limits to how accurately treatments could be tailored to groups? And is there a tradeoff between tailoring groups to treatments and treatments to groups?

[TwoCentimetre]

To be honest, these researches are far beyond my capability to understand. I am confused about how such deep learning methods retain the interpretability and economic meaning. And I do not understand the relation between the two paper to be presented. More particularly, in what aspects the second paper advances compared to the first paper.

[Yaweili19]

Thank you for sharing your work. I just started on Deep Learning and am finding some sections of your work, especially the interpretability, a bit hard to understand. Would you mind enlighten me on the choice of layer designs and how they matters in your interpretation?

[zhiqianc]

Hi Prof. Farrell. In your work, you mention that the adoption of machine learning methods can capture the potential heterogeneity. We know that traditional economic models may also have other problems such as endogeneity. Can the application of machine learning methods help capture and address these potential problems that are common in economic analysis?

[Sirius2713]

Professor Farrell, thank you for your sharing! Deep learning models are always reckoned as black-box models. And it's thrilling to read about your frameworks of inference based on such black-box models. My question is how can we generalize your framework to identify the causality in other deep learning problems like image/text classification?

[zoeyjiao1104]

Hi Professor Farrell, thank you so much for sharing your amazing work with us! You spend a lot of time explaining the models and the mathematics behind in the paper and there are a lot of eye-opening applications of the model. My question is in terms of causal inferences, do you think the nonparametric algorithm could contribute to any causal inference analysis?

[hazelchc]

Hi Prof. Farrell, thank you very much for sharing an inspiring piece of work with us! While your methodology can maintain the advantages of traditional economic models and can incorporate machine learning methods for estimating heterogeneity, how would you assess the limitations of your methodology? How might other social science disciplines benefit from the methodology? Thanks!

[fiofiofiona]

Prof. Farrell, thank you for sharing your work. From your paper, deep learning models and causal inference seem to be an innovative approach to many economic questions. How could future research extend from this framework to other economic (and in general, social science) problems? And how could researchers interpret the results from the deep learning model, given the black-box nature of it?

[linhui1020]

hi Prof. Farrell, thanks for your great piece in trying to improve the causal inference using deep learning method, as many scholars criticize deep learning is a black box and it is hard to be used for causal inference. I curious why how generalize the model would be? and the interpretation ability of model?

[zhiyun0707]

Hi Professor Farrell, Thanks for sharing your work with us! I find your work on deep neural network's use in econometrics fascinating. In the paper Deep Neural network for Estimation and Inference, in the empirical results section, you mentioned on the applications on US retailer of consumer products from a marketing point of view, which helped me to understand the theoretical parts better. Could you please give more real-world applications for your research results?

[yjhuang99]

Thanks for sharing your work! As you've stated, DNN had incredible empirical success in prediction problems, but you've managed to incorporate economic model inference with ML, which is pretty impressive and pioneering. However, it is still unclear to me how applied researchers could utilize your results when dealing with large-scale data. In particular, how can we empirically test the assumptions that are necessary to derive an unbiased estimator by structured DNN? Could you elaborate with an example? Thanks!

[LuZhang0128]

Thank you so much for these innovative papers. As I believe some social science people are more reluctant to adopt machine learning/deep learning models; I've encountered professors who asked me to explain what every layer of the ML model did. Do you think there's anything we can do to demonstrate the importance of ML in research? Do you think ML will eventually be a tool as important as traditional econometrics/statistics that all students need to learn?

[a-bosko]

Thank you for sharing your work with us! It was interesting to read about machine learning and deep learning methods can be used to find heterogeneity in the field of economics. I also enjoyed reading about the link between deep learning and neural networks While I am not an expert in this field, I always find it very interesting to read more about it!

In the conclusion of your paper titled "Deep Neural Networks for Estimation and Inference", you mention that there has been some resistance in applying deep learning methodology to the field of economics. Specifically, you mention that some of the reasons for this reluctance is due to lack of theory for use and interpretation. I was wondering if you could discuss more about why you believe deep learning methods are not being widely used in the field of social sciences? As computational social scientists, we often rely on machine learning and technical methods to answer social science questions.

[XTang685]

Hi Professor Farrell, Thank you so much for your work! Could you please talk a bit about the relationship between the two papers? I'd love to learn more about casual inference capabilities in deep learning tomorrow. Looking forward to seeing you!

[mintaow]

Thank you for sharing with us your research! I am particularly curious about the idea of recasting the standard economic model into nonparametric settings. Would you mind sharing with us the difficulties in developing this idea and are there any caveats when loading this method with empirical observational data (i.e. underlying distributions, etc)? Thanks again.

[yujing-syj]

Hi professor Farrell, Thanks so much for sharing with us! I am very glad to hear the application of deep learning in the economics field. Since I only have very entry-level understanding of the deep learning, my question maybe not that professional! I am wondering whether there are some tips or consideration for choosing the loss function, activation function and optimizer of the economics model? Also, do you think there will be large application of deep learning in the economics field as this method still is black box for most of people's understanding?

[YLHan97]

Hi Professor Farrell, Thanks for sharing your research with us. I have some questions as follows:

1. Since you have mentioned there is no additional derivations are required for the model, and the score can be taken directly to data, using automatic differentiation if needed. I am curious if there is derivations applied in the model, what might happen or is there any disadvantages in the process of model training?
2. In this article, you applied large-scale advertising experiment for short-term loans as the methodology. Would you please provide more real world examples in the relevant business industry?

[luckycindyyx]

Thank you so much for sharing such interesting work with us. Corporating causal inference methods into deep learning is an exciting idea. I do remember some scholars using "causal random forest" as a way of robustness check, and they seem to get more conservative result (with larger p values). Is "causal deep learning" also an effective way for robustness check? Also, I was wondering if you could share some further applications for causal deep learning. Looking forward to your presentation.

[ZHE-ZHANG-0213]

Hello, Professor Farrell Thank you so much for sharing! I am glad to hear about the application of deep learning in the field of economics. I wonder if this approach has any implications or considerations for the selection of loss functions, activation functions, and optimizers for economic models? You also mentioned that there is some resistance to applying deep learning methods to economics. Specifically, you mention that some of the reasons for this reluctance are due to the lack of theories to use and explain. I was wondering if you could discuss it more.

[ShiyangLai]

Thank you for sharing this interesting work. I note that you are trying to inject interpretability into deep learning models. This is very innovative for sure, though, I also know that a lot of other works on interpreting neural network models are emerging these days, some of which are very flexible because they do not need to be based on any existed economic models but still can provide reasonable explanations for the prediction outcome. I am thinking about how you want to compare your method with others. What are the merits and what are the weakness. Looking forward to knowing more about your thoughts.

[nijingwen]

hello, Professor Max Farrell. thanks for sharing your work!. In economics field, people normally use the past data to build the model to study the structural changes. However, the prediction is not accurate. You use the machine learning, and deep learning in large scale to build a neural network model which is more suitable for real world and real cases because the economics in real world could also been waved by eg policy, special event.etc. I am looking forward to learn how this method make the meaningful estimates and inferences from tomorrow's workshop.

[kuitaiw]

Dear Professor Max Farrell. Thanks for sharing your work. We know that deep learning directly predicts Y will lack interpretability. The first paper lets deep learning learn the parameters of an econometric model, and then uses the parameters to estimate Y. So I want to ask if using this method requires a lot of training data or something?

[fyzh-git]

Thank you Professor Farrell for showing us how the interpretability could be maintained when applying deep learning approaches to modeling individual heterogeneity. The combination with field experiment in the US retailer and the advertising experiment for short-term loans are inspiring. Could you illustrate these two applications more intuitively in presentation? Look forward to your talk.

[XinSu6]

Thank you so much for sharing your work! Its indeed insightful to see how machine learning methods can be used to in the work related to causal inferences and Econometrics. My question is did you have any major obstacles while developing these algorithms and methods? And how do you think your methods and applications can possibly be used in other fields?

Looking forward to your presentation

[yutaili]

Thanks for sharing your work professor Farrell. Your work in deep learning in the fields of economics is very impressive. My question about your papers would be that if this framework you developed in economics can be generalized to other disciplines somehow? Would like to hear your thought on that. Thanks.

[yhchou0904]

The article shows neural networks could be a well-performing model for us to identify treatment effects. I am just thinking about whether there is a possibility that we could estimate multiple effects at the same time with a single model just as we do with regressions? If it is the case, would we need a complex model, which would lower the interpretability, and a massive computation amount, which would increase the cost of evaluation? Does this trade-off really worth it?

[jiehanL]

Hi Professor Farrell, Thank you for sharing your work! I'm wondering, how would the two main challenges that machine learning algorithms have to face: the need for large-scale data and the problem with transfer learning, impact its functionality in causal inference? How would deep learning methods process, when the dataset is not big enough for algorithms to learn high-level causal variables from low-level observations?

[Toushirow1]

Dear Prof. Farrell, Thanks for sharing with us. I noticed that Professor Bruce Hansen recently proved that OLS is actually BUE so linearity is no longer to be the restriction for OLS. What do you think about Professor Bruce Hansen's work?

[cgyhumble0612]

Hi professor! First thanks so much for presenting us such hardcore and interesting findings. Actually, I start to use deep learning to cope with some daily work these days and find it useful to fit and estimate the model. However, as a student in econ, I just wonder how to interpret such complicated framework besides the output layer (layer and millions of links) inside the deep learning model in academic research paper. I hear a lot of comments related to the unexplainable feathers about that so I'm very curious about your opinions.

[chuqingzhao]

Hi Professor Farrell, thank you for sharing your work! I appreciate that you both introduce the idea of combining deep learning with traditional econometric model, but also you present interesting applications of your model. One question is while using deep learning in economic models will model assumptions be changed? Adding one point to my peers' question, I wonder what's the relationship between traditional econometric models, machine learning models and deep learning models? Do we know whether and how can we validate our results by combining different types of models in economic or social science research? Thank you in advance.

[NikkiTing]

Thank you for sharing your work! It was interesting to learn about how deep learning can be used to enhance structural models. My main question is since DNNs rely on large datasets, how can your proposed framework be applied to smaller datasets? Also, how do predictions using the framework compare to predictions using purely machine learning models with high predictive power but that are not interpretable?

[awaidyasin]

Hi Professor,

Thank you for your talk!

1. In your nonasymptotic analysis, you establish bounds for these neural nets using regression-type loss functions (in Theorem 1). Does this mean that the ability to make valid statistical inferences depends on the general class of loss functions, which may not be linear?
2. Methods like neural networks can involve substantial researcher discretion in specifying various aspects of the model (as you yourself show in the 8 network architectures on page 32). From a practitioner's perspective, what is the systematic way to decide upon an architecture? Do we need to run multiple Monte Carlo simulations before deciding on the one with minimum bias? (Alternatively, are there certain conditions that make the results less sensitive to the architecture used?)
3. From its application point-of-view, what advantage does this neural-net-augmented method give compared to more standard approaches in estimating average treatment effects in literature, given that neural nets are notorious for the computation required?

[Qlei23]

Thank you, Professor Farrell, for sharing your research with us. This machine learning framework is a huge enrichment to econometrics! As I understand it, the heterogeneity in the model is based on observed characteristics, how is it different from creating a special embedding layer in a standard economic model? Also, how does it connect to the studies in the field of heterogeneous macro? Thank you! Looking forward to your presentation tomorrow!

[sushanz]

Hello Professor Farrell, thank you for sharing your work with us today!

I am really interested in your topic of how deep learning is implemented on semiparametric estimations and treatment effects. Most of our classic approaches in finance or economy have metrics, for examples, MSE, MAE, R-square and etc to testify our models such as multivariate regressions. However, I wonder, just for my curiosity, how do we measure our deep learning models' consistency and concreteness? Are there any specific or customized metrics or loss functions for us in our future research as suggestions?

[Dxu1]

Thank you Prof Farrell for sharing your paper. I am curious about the issue with respect to curse of dimensionality, and wonder what would be the restrictions on the size of dataset if the deep learning based estimator that you propose were to be implemented.

[chentian418]

I am impressed with how this work combine the economic models and machine learning to to exploit rich data on individual heterogeneity. As this methodologies serves as the the first-step approach, I am curious about what are the functionality of the second and third steps. Does this methodology framework limit the application to economics models because of settings of deep learning and non-parametric methods, or in what way does it enrich the application to economic research? Thanks!

[xzmerry]

Thank you Prof Farrell! I think this is a really important research that incorporates deep learning neural networks into econometric research. It is amazing to see that we can conduct inference now under the deep learning based semiparametric method and how this method is useful under large scale data.

I am looking forward to hearing more in the workshop! And I am curious how this method could enrich econometric research related to causal inference and estimation of structural models.