Questions for Jake Hoffman on his 5/16 talk on "An Illusion of Predictability in Scientific Results"

[jamesallenevans]

Pose your questions here for Jake Hoffman regarding his 5/16 talk on An illusion of predictability in scientific results. In many fields there has been a long-standing emphasis on inference (obtaining unbiased estimates of individual effects) over prediction (forecasting future outcomes), perhaps because the latter can be quite difficult, especially when compared with the former. Here we show that this focus on inference over prediction can mislead readers into thinking that the results of scientific studies are more definitive than they actually are. Through a series of randomized experiments, we demonstrate that this confusion arises for one of the most basic ways of presenting statistical findings and affects even experts whose jobs involve producing and interpreting such results. In contrast, we show that communicating both inferential and predictive information side by side provides a simple and effective alternative, leading to calibrated interpretations of scientific results. We conclude with a more general discussion about integrative modeling, where prediction and inference are combined to complement, rather than compete with, each other. Contributing papers one and two.

[hchen0628]

Thank you for the sharing! In light of the findings on the illusion of predictability in scientific visualizations, how can researchers effectively communicate the limitations of their data without undermining the perceived validity and impact of their scientific contributions?

[MaxwelllzZ]

Thank you for sharing. Dr. Hoffman, in your researches, you've emphasized the importance of integrating explanatory (inferential) and predictive methodologies in computational social science to enhance the clarity and applicability of research findings. Could you elaborate on how this integrative modeling approach has influenced the methodologies and outcomes of your recent projects? Additionally, what challenges and opportunities do you see in encouraging broader adoption of this approach within the field?

[Caojie2001]

Thank you for sharing your interesting research! I wonder are there any example projects that you consider successful in integrative modeling, effectively combining prediction and explanation?

[Brian-W00]

The first paper revealed that displaying inferential uncertainty alone may lead even highly trained experts to overestimate treatment effects. How does displaying a combined visualization of inferential uncertainty and predictability of outcomes change professionals' understanding of scientific results?

[xiaowei-v]

It is a really interesting topic. The first paper shows people should be able to interpret the results more accurately with both inferential uncertainty and outcome uncertainty. And the second paper stress that we can provide people with more explainable results. This reminds me of some research on AI aversion. I would expect people demonstrate greater acceptance if they have access to explanations that help they understand the decision process. However I am curious how should we make some of the process say prediction explainable when it may require domain knowledge to fully grasp the mechanism.

[nalinbhatt]

In agent-based modeling there is ODD+D protocol to convey results and portray model complexity. Could having a set of agreed upon protocols to communicate results mitigate some of these issues.

[DonnieTang1]

Thank you for sharing. How does focusing on inferential uncertainty over outcome variability create misperceptions about the implications of scientific results

[Marugannwg]

I'm intrigued with the idea of further getting down the line of what "predict" means when many times we confuse it with finding significant evidence to support/explain a phenomenon under a (single) schema. From a researcher's perspective, what are some common goals that are overlooked? How do we tinker our research process toward particular goals? And what is this ideal landscape of research space when most studies follow the open-science guideline you posed in the paper?

[Ry-Wu]

Hi Dr. Hoffman, thank you so much for sharing your interesting work! I'm wondering if there are some specific challenges and potential biases that arise when scientific findings are presented primarily through inferential statistics?

[kunkunz111]

Thanks for your great work! I am curious about the implications of your findings for the broader scientific community. Specifically, could adopting a more nuanced approach to distinguishing correlation from causation enhance the reliability of scientific conclusions? Additionally, what challenges might researchers face in implementing such approaches, and how can we overcome these barriers to improve the accuracy and trustworthiness of research findings across various scientific disciplines?

[zihua-uc]

Thank you for sharing your work! I wonder whether there is significant heterogeneity of this effect across fields (e.g., economics, psychology, sociology)?

[WonjeYun]

Thank you for presenting your research. There has been research in various fields trying to explain the outcome of ML models. Especially, social scientists use causal inference as their main method. Do you think whether there needs improvement?

[bairr1208]

Thank you for sharing your inspiring research, Dr. Hoffman. Your study on the illusion of predictability in scientific results is fascinating and raises important questions. How do you think the awareness of this issue affects the perception of research conclusions among experts? Specifically, could certain communication practices serve as signaling tools to make observations or conclusions appear more believable across different scientific fields? In terms of policy, how might this illusion of predictability influence the way academic research supports decision-making and legislative action? Additionally, do you have any pedagogical advice for teachers and professors to help prevent such biases in school and college education? Lastly, how can the insights from your study be practically implemented to improve the accuracy of scientific communication in academic publications and presentations?

[vigiwang]

Thanks for sharing! How should we trade off causality and predictability in our research? Is there a reliable way for ML to guide us to understand the underlying mechanisms behind social behaviors? Or should we always use it to identify important features?

[binyu0419]

Thank you for your presentation, you highlighted the long-standing emphasis on inference over prediction in scientific studies and how this can mislead readers about the definitiveness of results. Could you elaborate on the specific methods or approaches that can effectively integrate both inferential and predictive information in scientific reporting?

[Daniela-miaut]

Thank you for your research! You mentioned in your paper that, by recognizing the limitation of predictive accuracy in social science, the presence of system complexity or intrinsic randomness can be yield evident. How do you think can we possibly integrate complexity with integrative models?

[xinyi030]

Thanks for sharing Dr. Hoffman! Considering the challenges in balancing prediction and inference, what practical steps can researchers take to ensure that their findings are communicated effectively, minimizing misinterpretation without oversimplifying complex data?

[lbitsiko]

Fascinating insights about the significance of inscriptions, immutable mobiles, visualizations in a sort of Latourian way. What would you think is the role of math education in misinterpretation?

[icarlous]

Thank you for your insights! You mention how computational social science could progress by combining predictive and explanatory modeling, but a key challenge is the cultural and methodological divide between fields focused on explanation and those prioritizing prediction, like computer science. How can we bridge these gaps to foster interdisciplinary collaboration? What practical steps or policies could research institutions or academic programs implement to promote integrative modeling?

[Aiwen-Xiao]

Thank you for sharing your insightful research. Given your findings on the misinterpretation of inferential uncertainty as outcome variability, how can researchers and practitioners better distinguish and communicate these concepts in their visualizations?

[cty20010831]

Thanks for sharing! I am wondering what practical steps researchers can take to avoid creating an illusion of predictability in their studies. How can educators and institutions incorporate your findings into scientific training and curriculum?

[Huiyu1999]

Thanks for sharing! My questions are: What practical steps can journals and researchers take to incorporate both inferential uncertainty and outcome variability in their visualizations to improve the clarity and accuracy of scientific communication? How can future studies build on the findings of this paper to further investigate the impact of visualizing both inferential uncertainty and outcome variability on various audiences?

[Yunrui11]

Thank you for sharing your insights through these abstracts. I found the concept of integrative modeling particularly intriguing as it suggests a balanced approach to incorporating both predictive and explanatory aspects of research. Given your extensive background in both academia and industry, how do you see the integration of prediction and explanation evolving in the field of computational social science, particularly with the ever-increasing availability of big data? Additionally, what are some of the practical challenges you foresee in achieving effective integrative modeling, and how might we, as emerging scholars, prepare to tackle these challenges?

[aliceluo1]

Thank you so much for sharing! Given the challenges associated with balancing inference and prediction in scientific studies, what practical steps can researchers take to ensure that their findings are communicated in a way that appropriately reflects both aspects, thereby reducing the illusion of predictability for both experts and lay audiences?