Questions for Jake Hoffman on his 5/16 talk on "An Illusion of Predictability in Scientific Results"

[jamesallenevans]

Pose your questions here for Jake Hoffman regarding his 5/16 talk on An illusion of predictability in scientific results. In many fields there has been a long-standing emphasis on inference (obtaining unbiased estimates of individual effects) over prediction (forecasting future outcomes), perhaps because the latter can be quite difficult, especially when compared with the former. Here we show that this focus on inference over prediction can mislead readers into thinking that the results of scientific studies are more definitive than they actually are. Through a series of randomized experiments, we demonstrate that this confusion arises for one of the most basic ways of presenting statistical findings and affects even experts whose jobs involve producing and interpreting such results. In contrast, we show that communicating both inferential and predictive information side by side provides a simple and effective alternative, leading to calibrated interpretations of scientific results. We conclude with a more general discussion about integrative modeling, where prediction and inference are combined to complement, rather than compete with, each other. Contributing papers one and two.

[yiang-li]

How might this emphasis on integrating predictive information influence future research practices in fields that traditionally focus heavily on inference, like sociology where inference allows for testing the theoretical frameworks?

[XiaotongCui]

Thanks for sharing! You mentioned that even experts might confuse inference with prediction. How should we assist them in distinguishing between the two more effectively during the process of education and training?

[zcyou018]

Thanks for sharing! my question is how can the integration of explanatory and predictive models in computational social science improve the robustness and reliability of conclusions drawn from large-scale data analyses, and what are the potential pitfalls or limitations of such integrative modeling?

[zimoma0819]

Thanks for sharing! The study highlights a crucial gap in scientific communication by demonstrating that the exclusive focus on inferential uncertainty often leads to overestimations of treatment effects by experts. Incorporating both inferential uncertainty and outcome variability in data visualizations could help foster a more accurate understanding of scientific results among professionals. This approach could potentially revolutionize the interpretation of research findings, especially in fields where precise data interpretation is critical. My questions are whether this approach could become a standard practice and what we need to do to persuade people to adopt this approach to foster a good environment to present more accurate understanding of scientific results.

[bhavyapan]

Thanks for sharing your work! I am wondering about the implications of this awareness when it comes to the perception of research conclusions -- would experts be more likely to find certain observations or conclusions from research "believeable" if certain practices in communicating them are followed -- essentially could such approaches be possibly used as signalling tools in academic? How prevalent could this issue be across different scientific fields? In terms of policy, how could this illusion of predictability affect the manner in which academic research supports decision-making or feeds into legislative action?

[Dededon]

Thank you for the sharing! I’m curious in your insights about combining machine learning with causal inference research. How do you think of the theoretical frameworks as CausalBERT, that seems to sacrifice interpretability towards better estimation?

[Adrianne-Li]

Hello Dr. Hoffman,

Thank you for your insightful exploration of the illusion of predictability in scientific results. Your discussion raises important questions about the perception of research conclusions. Could the strategies you've proposed for communicating both inferential and predictive information serve as signaling tools within the academic community to enhance the believability of research findings? Additionally, how widespread do you believe this issue might be across different scientific disciplines? From a policy perspective, what impact might this illusion of predictability have on the way academic research is utilized in decision-making processes or influences legislative actions?

Looking forward to your thoughts, Adrianne(zhuyin) Li (CNetID: zhuyinl)

[kiddosso]

Thanks for sharing Dr. Hoffman! Your research is really inspirational. I wonder if you have pedagogical advice for teachers and professors in schools based on your paper An illusion of predictability in scientific results? Such a bias may be prevented in school and college education.

[oliang2000]

Hi Dr. Hoffman! In a class, we read a quote by John Rogers Searle that said, 'Prediction and explanation are exactly symmetrical. Explanations are, in effect, predictions about what has happened; predictions are explanations about what’s going to happen.' In your paper, you attempt to distinguish between these concepts. I'm wondering, would you say they are continuous/symmetric concepts or how do you think they relate to each other on a high level?

[C-y22]

Thank you for sharing! You discuss the potential for computational social science to advance by integrating predictive and explanatory modeling. However, one of the major challenges you highlight is the cultural and methodological differences between fields that traditionally focus on explanation and those that prioritize prediction (like computer science). Given this context, how do you envision overcoming these differences to foster collaboration between these disciplines? Specifically, what practical steps or policies could be implemented to encourage integrative modeling in research institutions or academic programs?

[66Alexa]

Thanks for sharing! What areas of future research do you believe are crucial to further understanding and addressing the illusion of predictability in scientific results?

[yuy123337]

Hi Dr. Hoffman! I am wondering why is prediction important in scientific research, particularly in the context of computational social science? Based on the discussed integration of predictive and explanatory modeling in computational social science, why is there an increasing emphasis on predictive modeling? What are the strengths of predictive approaches, and how do they complement traditional explanatory methods?

[Zihannah11]

Thank you for sharing. How do you think the emphasis on inference over prediction in various fields has influenced the way scientific studies are conducted, reported, and interpreted by both experts and the general public?

[natashacarpcast]

Hi! Thank you for this interesting research. I'm wondering which implications does this have for previous science? Would you say you're skeptical of everything that's already done? Or are there any areas where these problematics wouldn't be so harmful?

[iefis]

Thanks for sharing your research! Could you provide some more concrete examples that apply the integrative modelling approach in social science research?

[HamsterradYC]

Thanks for sharing your work! I'm wondering if the experiments described in the paper were conducted in controlled environments using hypothetical scenarios, can these scenarios accurately represent the complexities and challenges of everyday scientific communication?

[zhian21]

Thank you for sharing the interesting work. It explores how visualizing only inferential uncertainty in scientific findings can lead to misperceptions about the predictability and importance of these results. Through three randomized experiments, the study demonstrates that showing both inferential uncertainty and outcome variability leads to more accurate perceptions of scientific findings. The research highlights the need for better visualization practices to avoid overestimating the effects of scientific treatments and results. I am curious how the insights from this study can be practically implemented to improve the accuracy of scientific communication in academic publications and presentations.

[AnniiiinnA]

Hi Dr. Hoffman, thanks for sharing this interesting finding! In "Integrating Explanation and Prediction in Computational Social Science," you argue for the necessity of combining predictive and explanatory modeling to advance the field of computational social science. What specific barriers do you think contribute to the relative lack of integrative modeling research in computational social science? What strategies might be most effective in overcoming them to encourage more research that integrates predictive and explanatory approaches?

[Jessieliao2001]

Thanks for your kindly sharing! My question is : How does an emphasis on inference over prediction in scientific studies contribute to an illusion of predictability, and what are the implications for both experts and the general public in interpreting scientific results?

[JerryCG]

Dear Jake,

This is a very interesting statistically related pychological phenomenon. Indeed, as the sample size increases, the estimate, for example, the sample mean, will decrease greatly in the SD. The inferential uncertainty drops a lot. However, the SD for the outcomes should be relatively stable since they come from the same distribution. We are likely to be cheated by the seemingly significant difference in estimates and over-relies on this when predicting individual outcomes. The visualization does play a big role about how to form the correct understanding.

I wonder whether we can develop a new measure of the estimates in terms their predictability, adjusting the variation in the outcomes?

Best, Jerry Cheng (chengguo)

[yunfeiavawang]

Thanks for sharing! The study reveals that focusing only on inferential uncertainty can cause experts to overestimate treatment effects. Could this approach become standard practice, and how can we persuade people to adopt it for better scientific communication?

[volt-1]

Thanks for sharing your insightful findings. In your research, you've shown how the way we present scientific data can affect how people understand the results, particularly when showing both the uncertainty of the estimates and the variability of individual outcomes. Could you discuss how using complex statistical methods like Bayesian hierarchical models, which show both these aspects, might change how people perceive scientific findings? Also, how might using these methods more regularly in reporting results impact public trust in science, particularly in fields with high variability like epidemiology or social science?

[shaangao]

Thank you for sharing your research! Given the results, what efforts can we make to support research groups with limited resources to better estimate individual treatment effects, on top of the traditional focus on average treatment effects?

[Kevin2330]

Hi Professor Hoffman,

I found your research fascinating, particularly regarding the balance between predictability and inference/interpretability, which is an area I'm also keenly interested in.

Question for 1st paper: How do you think visualizing both inferential uncertainty and outcome variability can be implemented in standard scientific reporting to minimize misperceptions among various audiences?

Question for paper 2: What are some practical challenges you foresee in integrating prediction and explanation within computational social science, and how can researchers overcome them?

Could you elaborate on the schema you proposed for thinking about research activities along the dimensions of causal effects and predictions? How can this schema guide future research directions?

Thank you!

[jiayan-li]

As computational social scientists, how can we reconcile the need for clear communication of inferential uncertainty and outcome variability, as highlighted in the first study, with the dual focus on prediction and explanation, as advocated in the second paper? Specifically, what strategies can we employ to ensure that our visualizations and models not only accurately convey the uncertainty and variability in our data but also effectively integrate causal explanations and predictive power to enhance the overall interpretability and utility of our findings?

[yunshu3112]

Hi Dr. Hoffman, I am very interested in your findings in your paper "An illusion of predictability in scientific results". I wonder how you interpret the tension between causality and predictability. In addition, do you think future improvement of statistical tools can mitigate this illusion of predictability by enhancing inferential certainty?

[adamvvu]

Thanks for sharing your work. It highlights the importance of interpreting the point estimates and uncertainty in the context of the outcome's variability. I wonder however if the results from the experiments were more of a cognitive bias related to scale? Were all three experiments on the same scale?

[Weiranz926]

Thanks for your sharing! Given the emphasis on presenting both inferential and predictive information to provide a more calibrated interpretation of scientific results, what practical steps or guidelines would you recommend for researchers to ensure their studies effectively balance these two aspects, especially in fields where prediction has traditionally been undervalued?

[Yuxin-Ji]

Thank you for sharing! It's very interesting to learn about this psychological phenomenon on how people perceive the effect when presented with different visualizations. Do you think there are any con side for presenting the inference and prediction together, such while they avoid overestimating the effect, people are less confident in reaching a conclusive finding?

[yuzhouw313]

Hello Professor Hoffman, Thank you for sharing your work with us. While I understand that inferential uncertainty and outcome variability focuses on inference and prediction respectively, where the former draws conclusion from sample to population and the latter predict future events given current/past events, are there special cases of the overlapping of these two? Specifically, could you elaborate on how both inferential uncertainty and outcome variability impact the reliability and validity of conclusions in longitudinal experiments where both internal and external factors play a significant role over time?

[kexinz330]

Thank you for sharing your research! I am interested in knowing how can scientists effectively communicate both inferential and predictive information side by side? Are there any best practices or templates you recommend for presenting statistical findings?

[h-karyn]

I love the second paper, as it touches on many questions I had about the prediction vs theory debate in computational social science. The authors argue that prediction and explanation are complementary rather than competing goals. How do they propose to handle the potential conflicts that might arise when a model's predictive accuracy decreases due to the inclusion of more interpretable causal mechanisms?

[ZenthiaSong]

Hello, thank you for sharing! How generalizable do you think your findings are across different scientific disciplines? Are there particular fields where you expect the illusion of predictability to be more or less pronounced?

[ecg1331]

Thank you so much for sharing your work! It was a a really interesting point you made with your research in "An illusion of predictability in scientific results", and it seems like visualizing the individual outcomes would make a large difference in the correct interpretability of data in the scientific community. Since this (for the most part) is an easy remedy to a problem, do you see it being adopted by the community?

[beilrz]

Thank you for sharing this. I agree with the first paper's finding that researchers should aim to present a more comprehensive picture of their result (data range) rather than reducing it to a single-point estimator. The second paper highlights some challenges in the field of CSS, especially the conflicts between aiming for prediction accuracy vs aiming for explanation. In your opinion, what are some paths forward for CSS? given the increasing complexity of current models in the field, and lacks of human insights into them.

[jingzhixu]

Thanks for sharing this inspiring work. I've learned a lot about arranging the visualizations to better tell the story in my future research. Just one quick question, if the graphs were shown to the interviewee with description of the graph and results, would the misinterpreting problem be that significant?

[MaoYingrong]

Thanks for sharing your work! My question is: what broader implications do the results of this research have for the field of scientific inquiry in the social science area? How integrative modeling approach can be taken in a specific field?

[HongzhangXie]

In "Integrating explanation and prediction in computational social science," the author divides empirical modeling into four quadrants. It is mentioned that research focusing on predicting outcomes under interventions or distributional changes is very limited, indicating a scarcity of studies that simultaneously focus on both predictive power and explanatory capability.

I am curious whether we can, through certain experimental designs or research strategies, obtain research outcomes guided by the three different approaches of Quadrant 2: Explanatory modeling, Quadrant 3: Predictive modeling, and Quadrant 4: Integrative modeling. By comparing the predictive accuracy and explanatory power of the three models, we aim to determine whether it is possible to achieve high accuracy and high explanatory capability simultaneously in Integrative modeling.

[jialeCharloote]

Could you discuss any challenges or limitations you encountered in conducting your randomized experiments to demonstrate the effects of presenting inferential and predictive information side by side?

[Hai1218]

Thank you for coming to our workshop! How do you address the challenges of inferential uncertainty and outcome variability in predictive algorithms to ensure they maintain accuracy and fairness, especially when these algorithms are applied different population? How does the approach of shifting focus on outcome variability change our current understanding of accuracy-biased tradeoffs?

[PaulaTepkham]

Thank you so much for your two interesting paper! For me, increasing the effectiveness of communicate is one of the most important points that will drive academia in more developed level. Can you provide examples of how improved communication of both inferential uncertainty and outcome variability might change the interpretation of results in these fields?

[nourabdelbaki]

Hi Prof. Hoffman, thanks for presenting your work!

In light of the findings that visualizing both inferential uncertainty and outcome variability leads to more accurate perceptions of scientific results, how might this approach influence the broader scientific community's practices in communicating research findings? Additionally, what challenges might arise in implementing this approach across various disciplines, and how could researchers address these challenges effectively? And I guess, how would these approaches differ based on whether this is for public dissemination of findings VS within the scientific community?

[lim1an]

Thanks for sharing! This might be a little off the research topic itself, but when we try to communicate the results to the external parties with less background in this, how would you balance the accuracy and understandability of the statistics results?

[ethanjkoz]

HI Professor Hoffman, I found the paper on the illusion of predictability in scientific results a very interesting read. How does this work recontextualize the increasing mistrust of science more broadly (upwards trends of climate change denial, anti-vaxx, etc). How can we adequately disseminate knowledge given this increasing amount of distrust and "skepticism" in the scientific community?

[franciszz992]

Your papers are very inspiring and interesting to read. Can't wait to see the presentation tomorrow. I'm convinced of the suggestions that you set out in the Perspective paper, and I'm particularly interested to see your argument to use a common task framework to centralize the collective efforts in a research field. Has it ever been sucessfully implemented? Who should be centralizing the research? And does it make (political) economic sense for such a centralized framework/institution to exist?

[yuanninghuang]

Could you elaborate more on the specific methods or formats that you have found most effective in achieving the balance of presenting both inferential and predictive information side by side for more calibrated interpretation of scientific results, essentially in communicating with experts and non-experts in the field?

[ksheng-UChicago]

Thanks for sharing. The research result is very interesting. I wonder if such research can be helpful in both academia and industry. Definitely would like to learn more about it.

[Joycepeiting]

Thank you for sharing! Paper 2 mentions how structural models in economics are positioned in quadrant 4, Integrate modeling, as they are capable of conducting predictive counterfactual analysis. However, many of these models rely on specific theoretical and statistical assumptions. What are your thoughts on the tradeoff between the flexibility of these underlying assumptions, which may impact generalizability, and the predictability built on causal statements?

[YutaoHeOVO]

Hi Mr Hoffman,

I think the key of your argument is on inference versus prediction. The emphasis of inference actually originates from the requirement of yielding causal interpretations instead of simply predicting the dynamics. (Researchers tend to care more about what is behind the data instead of what can be directly observed from the data.) Indeed, it can be the case that some causal findings might be trivial, yet I am afraid your concern on predictability and misperception in scientific finding is simply emphasizing the trivial facts that many social scientists, or at least economists are already well acquainted of.

[lguo7]

Thank you for sharing your research! In the discussion, you suggest that displaying both inferential uncertainty and outcome variability can reduce the “illusion of predictability.” Considering the practical constraints of visualizing large datasets or highly skewed data, what alternative visual representation strategies would you recommend to effectively communicate these statistical concepts?