01/23: Wang

[jmausolf]

Comment below with questions or thoughts about the reading for this week's workshop.

Please make your comments by Wednesday 11:59 PM, and upvote at least five of your peers' comments on Thursday prior to the workshop. You need to use 'thumbs-up' for your reactions to count towards 'top comments,' but you can use other emojis on top of the thumbs up.

[anuraag94]

Thank you in advance for your presentation, and for the interesting read. My question is about the interpretability of the different modeling decisions you made across your two articles.

In "Early-career setback and future career impact", you use a regression discontinuity design, which is well-motivated by the fact that you're observing behavior and outcomes around a threshold. This has a rich tradition within the econometrics literature, and you're able to make a clean case with strong explanatory power.

In your later Nature article, you use a single parameter phase transition model. Yet, this is still intuitively motivated, since each of the three cases you consider have good analogies to physical phase transition.

Why did you decide to use methodologies from two very different literatures to tackle similar questions? Do you believe that one is more intuitive, or has more explanatory power than the other? How would you suggest that we examine the results of these side-by-side?

[SoyBison]

Thanks for coming to our workshop! My question has to do with future directions for your research. In Early Career Setback and Future Career Impact there is some mention of heterogeneity between the group that is negatively impacted by an early career setback and the group that is positively impacted. Have you seen any preliminary results that suggest what this heterogeneity might look like should it exist?

[policyglot]

Dr. Wang, You read our minds! Or mine, at least. I've unfailingly been wondering about failure for years. When does one call it quits and toss in the cliched towel? When is it too severe to persevere? I wish I'd known that I just had to check for an empirical power law to feel powerful again.

Since your research hails from a business context, let's take a business spin on this. Many companies known for their innovations are often not the first to come up with the idea. (Originality is the art of concealing the sources. Not my original quote). Zuckerberg wasn't the first to conceive of social networks. Google wasn't the first search engine. Long before Google lapped up Youtube, the latter was conceived as an online dating site.

So in terms of the parameters of your k-alpha model, how would you capture these two differing sets of dynamics: 1) Facebook, Google- Learning from someone else's initial failures 2) Youtube- Learning from one one's own errors but pivoting into a completely new business model

[liu431]

In your Nature article, you chose science, startups, security as three fields to study the dynamics of failure. However, the cost of failure for each is very different. If researchers don't get an NIH grant, they can apply to others and try again next year. In comparison, if a startup fails, it means many people lose their jobs. If a terrorist organization fails an attack, their risk of being caught increases. So the question is how to distinguish the dynamics of different fields and also should we consider the potential cost of failure into the model?

[nwrim]

Thank you in advance for the presentation and a great read! The article Early-career setback and future career impact was very interesting to me and I have a question regarding the article:

Did you find any significant differences between the researchers who got attrited after the near misses and who survived after near misses? I think this is a very important question especially regarding that the science communities are trying to build a system that allows as many talented researchers to survive in academia. The first thought that passes my mind is the difference in funding - as the very impressive Science magazine article Another tenure-track scientist bites the dust illustrates:

But for Matthew, this wasn’t just another grant review cycle. He’d been an assistant professor for 6 years. In that time, he’d taught classes, published papers, spoken at meetings, conducted research, and trained students—but, despite submitting proposals during at least a dozen review cycles, he had earned exactly zero grants. Matthew had used up his startup funds. If he wanted to remain on the tenure track, this was his last shot.

I had read through the supplementary information of the article and realized that it is very difficult to get a grasp of how much funding each PI has, but I was wondering if you were able to find a funding difference between the people who disappeared in the dataset later and the people who were able to carry on their academic endeavor. Although I had stressed funding because this research was primarily based on NIH funding, but any other interesting difference you find will be delightful to hear!

[Yilun0221]

Thanks for the presentation! Your research analyzed what failures are more likely to lead to success. My question is, even if we know this, what other application value does it have besides predicting whether it can succeed based on failure experience?

[WMhYang]

Thank you very much for your presentation in advance. My first question is a technical question regarding your Nature article. How did you choose the range (-5, 5) of the normalized scores as your sample of interest? In addition, in your fuzzy RD analysis, is the range (-5, 5) the same? My second question is about the link between two articles. I wonder whether the researches of the narrow winners show more early sign of ultimate failure rather than ultimate success compared with the near losers. Is there any possibility that we could test it?

[timqzhang]

Thank you for your presentation, this topic is truly closed to students as we may face various failure in the beginning career. My question is that, how to claim that the case of NIH system could represent the general impact of setback on the future career development? In other words, as the junior scientist is not a quite common identity for young people, it may be hard to generalize this conclusion to the common people. I am quite excited to see if this model could extend to more fields and see whether it is widely applied.

[ziwnchen]

Thanks for the interesting presentation! My question is similar to @timqzhang, as for using funding failure as the indicator of setback. From previous research, we know that It is possible that funding may have certain "taste"(e.g., hot topic or innovative idea). Therefore, using the evaluation score by the funding committee to measure junior PIs, and using the result of the funding application to represent setbacks could both be "biased" to some degree. If it is the case, will such "bias" influence the conclusion? Or the generalization ability?

[wanitchayap]

Thank you for the papers! I really like your paper on Early-career setback and future career impact. I think the analysis is very well-designed and robust!

I have questions on long-run grants and applications for grants. You mentioned in the paper that you accounted for NIH, NSF, and other grants in the first 5 years, and that "near misses did not acquire more funding than narrow wins". However, it is unclear to me what happened in year 6-10 and how situations on grants in the last 5 years affect the performance.

I am also wondering if considering not only the net amount of grants each scientist won but also the proportion of times they won the grants (that is #_won_grant/#_apply_grant matters?. From your other paper, we can see that not only one failure matters but also the whole sequence of failings (and/or winnings) matters. I especially wonder if the proportion of winning the grants can, to some extent, explain who stayed and who quitted in the near misses.

[ShuyanHuang]

Thanks for presenting. In your model of failure, each attempt consists of many components, with each component being characterized by an evaluation score. This formulation assumes these components are clearly defined and can be evaluated without any uncertainty. You also assume that the weights of the component are equal. However, in real life, the ability to decompose tasks and evaluate each component of past attempts are considered key factors of success. And both the evaluation scores and the weights are often unknown to people. Maybe extending the model by explicitly accounting for these uncertainties and decision-making processes will shed more light on this topic.

[hihowme]

Thanks for your presentation and bringing out this interesting paper! In your paper of Quantifying the dynamics of failure across science, startups and security, you made comparisons between 3 different issues, 'NIH Grants', 'Startups', and 'Terrorist Attacks' and there seems to be a consistent result. However, those 3 things are all kinda big issue, my question is that do you think your model could be applied to some normal and popular goal, like regular training, diet and find a good job?

[romanticmonkey]

Thank you for your presentation! This is a truly warm paper for everyone working hard in the academia. I noticed this research is based on the data in NIH, so I wonder does the result applies to all academic disciplines? Will different field’s distinctive culture cultivate different success/failure motivation mechanisms?

[anqi-hu]

Thank you for sharing your work with us. Regarding your paper Early-career Setback and Future Career Impact, do you think the results would look different if instead of a quantitative measure of success (i.e. obtaining a grant), a qualitative measure of career success is applied? I think it might be helpful to incorporate qualitative measures, especially when comparing the magnitudes of success/failure in the successful and unsuccessful groups. Are there particular reasons to prefer quantitative measures in this study?

[linghui-wu]

Thank you, Dr. Wang, for the interesting paper "Early-career setback and future career impact" and the encouraging conclusions. My question is very similar to that of @liu431 regarding the different costs of failure. Not only between academia and industries but also among different disciplines, we can observe various costs along the path to success. In the paper, data on junior scientists applying for NIH R01 grants were collected, I wonder whether the results remain robust if we alternatively focus on scholars concentrating on other disciplines, say humanities? If not, how would you modify the model in order to take into consideration that?

[ZhouXing-19]

Thank you for your presentation, Dr. Wang. I am very interested in your paper "Early-career setback and future career impact", but I am a bit confused about how "early" is defined since even within the same discipline, the time span for research can vary largely. Also, I am curious about the heterogeneous impact of early-career setbacks among different subjects, especially the distinction between natural science and social science.

[zeyuxu1997]

Thanks for your presentation. I have several questions. Firstly, do you have any advises to those who want to replicate your research in different fields? The result of first application to NIH funding is a great representation of early career success/failure, but there may not be such convincing and accessible representations in other fields. Secondly, what's the most significant factor contributing to the success of this research? Finally, will the conclusion hold if we extend the range of objects to more junior PIs, for example, those whose normalized scores lie with in the range of (-50,50)? I understand your design aims to make near misses and narrow wins similar to each other except for final results, but consider those with a greater failure and those with a greater success might make the conclusion more convincing.

[lyl010]

Thanks for your presentation! I am surprised by the result of 'Early-career setback and future career impact' that failure in early stage can boost better performance later. And I am wondering why failure can be so powerful, so as @Yilun0221 said, could you please introduce what other effects can early failure result in? Maybe for those who survive after the failure, do them have longer career? Besides this, I think the time that a scientist spend in that field can help to understand the success in long term career. Scientific community may change its interest through time, and in fact, we could not really predict the future research interest based on our present knowledge. Does the latter success due to the fact that scientific community change its interest and begin to appreciate the work that it did not appreciate at the beginning?

[chiayunc]

Thank you in advance for the presentation. My question is regarding the concept of success and failure in life and to what degree does getting a grant or not reflect that. Early-career setback and future career impact draws on the notion that what doesn't kill you make you stronger, and did find a discrepancy between the survivors in the near-miss group and the rest. But in real life, applying and getting a grant is arguably only a way to secure resources. The applicants know that getting the grant or not is not solely based on the merit of their research or their credentials. If as an applicant, I don't get the grant, I will just find elsewhere for funding. If they know that getting a grant or not is not completely dependent on their performance, and it is partly a bureaucracy, not getting the grant would hardly be a failure they need to survive.

[lulululugagaga]

Thank you for your presentation. Your research on career setback is really interesting, however, I still have some questions about how you may conceptually define 'early failure'. To be more specific, 'early' may point to different life period for people in different fields and 'failure' can be quite big or small. Do we see all disciplines as the same or there are some other measuring factors that shall be discussed?

[keertanavc]

Thank you for sharing interesting research. The methodology used in this study is mostly quantitative data. I was wondering how the paper could have been extended using qualitative information about researcher characteristics like mentorship, grit etc. which like you've mentioned haven't been measured. What strategies could be employed (e.g interviews) and how to further support your hypothesis? Thanks!

[MegicLF]

Thank you for bringing out this interesting topic! As you mentioned in both papers, early failures in the career may have a positive impact on people's growth, and you used data from NIH to discuss the failure in academia. I wonder if your model will also work for other fields such as engineering or humanities. Would that be a totally different story? Is it possible that there exist cases that may demonstrate a totally opposite conclusion to your model? If so, what characteristics those cases may have? Thanks!

[vinsonyz]

Thank you for your presentation, Professor Wang! The setting of the research is really interesting! My concern is how would you treat the heterogeneity issues, how would this setback effect differ in variaous fields?

[tonofshell]

Thank you for sharing your work! I was wondering if you think that current science programs in higher education are effectively integrating failure into their curriculums? How might a program improve their curriculum based on the findings from this research?

[ydeng117]

Thank you for the presentation. It is encouraging to find that making mistakes could eventually lead to successful outcomes. You pointed out that failures at the early stage can signal potential successful outcomes even without the distinguishing features. Can individual scientists mimic the pattern to approach their eventual successful findings? or we can only use this mechanism to determine which mistakes worth investing in? Can your findings be generalized to other vocations?

[HaowenShang]

Thanks for your presentation! For the paper talking about dynamics underlying failure, it’s a really interesting finding that early-career setback appears to cause a performance improvement among those who persevere. I am quite confused whether this finding can be generalized in other fields rather than just in academia?

[YuxinNg]

Thank you for the readings. In the article "Quantifying the dynamics of failure across science, startups and security", you did research in fields of science, startups, and security. These three fields seems very irrelevant from each other. I am wondering why you chose to look into these three specific fields? Have you also tried to apply your methods and models to other fields, for example students' academic achievements, job applying, etc. Do you think what you found in the article applies to all the other fields? Thanks.

[goldengua]

Thanks for your presentation! I like your paper about how early failure in academia could predict future success or quit. I am especially interested in the fact that those who survived the setbacks did better in their future career. There are certainly some psychological motivations for that, and I was wondering how we could dig into the underlying reasons in psychological aspect with the existing data？

[hanjiaxu]

Thank you very much for presenting such an interesting topic! This is such a relevant topic for graduate students who want to become future scientists. In addition, the science of "failure" is such a good topic but has been less mentioned in training. What I am concerned, though, is to overgeneralize "NIH attrition" as a standard for success and failure. Admittedly, for PIs, NIH attrition is a critical standard to evaluate their career. However, there are many people might take alternative trajectory, but not necessarily mean that those are failures.

[sanittawan]

Thank you in advance for sharing your research with us. I learned a lot from your paper published in Nature Communications. The use of regression discontinuity design in this context is ingenious. I have two questions while reading the paper.

(1) If you have to guess, do you think that your result is generalizable to researchers in the social sciences? I am aware that without the data and conducting a research in the context of social scientists, it would only be speculation and perhaps not a fair question to ask. However, I am wondering if you came across certain characteristics that would be similar for social scientists which suggest that the result from the natural sciences might hold true in a different context too.

(2) I am new to matching, but I am curious how you chose coarsened exact matching over other available matching methods. Is CEM always preferable to normal matching?

[ShanglunLi]

Thank you for providing such an interesting paper to read! I see 'early failure' mentioned in the paper. My question is how you define this term. Since early and failure are all sorts of qualitative variables, do you have a quantitative boundary to help clearly define it?

[YanjieZhou]

Thanks very much for your presentation! Scientific failure does play an imporant part in our research. But from my perspective, almost every research will be carried out in an direction that diverges from the original track. So I am curious what your opinion is on the pervasiveness of failure like setting a more precise measure to quantify this concept further.

[dongchengecon]

Thank you so much for the presentation about the dynamics of failure. And it is great that the model could offer some implications of the early signals. Could you explain to us more in detail about the ways more complex mechanisms could be incorporated in your model? Besides, is there any reduced form measurements that could be created from your model and serve as the signals for the agents to behave accordingly and select a more success likely path?

[PAHADRIANUS]

Thank you for the two exquisitely crafted pieces that quantify and model the notions of achieving success. While some of the arguments proved by the statistical tests, such as how repeated failures could develop a pattern towards success or Nietzsche's statement on how near-miss makes one stronger, are well-known and accepted social norms, to explore the mechanism behind them is truly the great contributions of the studies. I am also truly impressed by the thoroughness of the robust checks of the two papers. Frankly upon seeing the title and skimming the abstracts I was quite dubious about how reproducible and externally-valid the tests are, but I was then much more convinced having read over how you varied multiple parameters and add in some stochastic features to prevent the findings from being singular cases. I am just curious about one thing: are the young scholars in the near-miss paper aware of how close they were to the NIH grant threshold? I am aware that you selected near misses and narrow wins based on the ranking of the evaluation scores; are these scores and the cut made public or informed to the applicants?

[yutianlai]

Thanks for your presentation! Dr. Wang, your research Early Career Setback and Future Career Impact appeals to me a lot. I’m wondering how you would like this research to be applied in reality. Many questions here have asked the detailed definition of “early” and “failure”. After specifying the two terms in different fields or different scenarios, what social change would you like to expect? For instance, to tailor failure education for different disciplines?

[chun-hu]

Thank you so much for coming! The research on early-career setback and future career impact is truly inspiring. I might have missed this, but how did you define "narrow-wins" and "near-misses" in this research? Also, is it possible that different institutions have different criteria for funding eligibility, so that "near-misses" for NIH might be able to get grants from other places?

[bjcliang-uchi]

My question is, in general, what constitutes a failture/ setback when we want to generalize this model, and will a loose definition influence the difference between stagnation and progression as shown in this model? Also, does time (particularly, the discounted memory of past history) influence this model?

[SiyuanPengMike]

Thanks a lot for your interesting and inspiring papers. I'm quite impressed by your findings. However, I'm still a little bit confused about the relationship between failure and success. Should failure and success in the same domain to make sure they have an inherited correlation that has been found by your models? For example, will a failure in the biochemistry industry contribute to the success of an electric vehicle startup?

[tianyueniu]

Thank you for presenting these interesting papers! The regression discontinuity design is very inspirational. My question is, other than paper publications and citations, are there any other ways to quantify these scientists' success in the long run? (e.g. If rejected junior scientists turned to the private sector, is there any way to quantify their success and compare them with those who got the grant?)

[heathercchen]

Thank you so much in advance for presenting such an interesting topic for us! I have a question regarding the definition of "long-term effect" in one's academic career. Although it seems to outsiders that ten-years is a quite long time period, but is it sufficient to cause "long-term effect" when it comes to academic researchers? As we know in many fields of scientific researches, 10-years can be just halfway through a project.

[JuneZzj]

Thank you for presenting. It was a really interesting topics in causal inference. I have seen the stability analysis in different time period measurements. Why do you think it is important to include this analysis. Also, can the relation between the paper publication and the setback in the early research career be generalized in different fields of studies? Thank you.

[fulinguo]

Thank you very much for your research. In the paper, you show how future performances differ between the near-miss and the narrow-win individuals. If we assume that these two groups, on average, are at the same location of the distribution of total population's characteristics, the results in the paper show how setbacks impact applicants who are in this particular area of the distribution (the place around the winning threshold). I was wondering whether it is possible to examine the effects of setbacks on individuals in other places of the distribution. For example, if some very strong applicants lose because of bad luck, or if some poor applicants win by accident, could we examine how setbacks affect these kinds of people's future performances? Thank you so much.

[KenChenCompEcon]

Thanks for your presentation. This is a very intriguing and inspiring finding. I am wondering if any ideas can be put on the design of fund-giving systems that helps to shape more developed screening criteria? Could temporary setbacks play a role in raising scientific productivity? Thanks!

[jtschoi]

Thank you in advance for your presentation. I was astounded by the research design and results in the paper Early-Career Setback and Future Career Impact as well as the quantitative approach in the paper Quantifying dynamics of failure across science, startups, and security. Across the two papers, however, I was wondering whether, among those who barely miss the mark, there is any dispersion in the frequencies or varieties of applications to grants that the corresponding individuals make. I wonder if those who ultimately succeed in the long run (but fail in the short run) become more accustomed to the grant application process or devising more elaborate / convincing applications as opposed to those who succeed in the short run. Is there any information that you can shed on this? Thank you again.

[Leahjl]

Thank you in advance for your presentation and the interesting topics. I'm interested in the "failure dynamics" and "future career impact" you've mentioned in your second project. Does this model can be applied to other fields? For example, is there any relation between the current failure of seeking internships and later success in finding a full time job?

[hesongrun]

Thank you so much for the wonderful and encouraging research! Failure can also mean opportunity to reassess one's weakness and make progress to succeed in the future. My question is how do you address the endogeneity problem when you compare the long-run performance? People who decide to stay despite of failure may possess strong drive or ambition in the field. It may not be the failure that make them become more successful, instead, their success may be attributable to the inner ability or strong belief about future success. In another word, there may be survivorship bias in your research design. How do you address these issues? Thanks!

[sunying2018]

Thanks for your presentation. It is a fantastic finding on the relationship between early career setback and future impact. But I am curious about how to validate this relationship, in other words, how to exclude the influence of other factors such as luck and opportunities? Thanks!

[Anqi-Zhou]

Thank you for your interesting presentation in advance! How would you consider the difference between various industries? For example, some industry in which age matters a lot. Also, as some classmates mentioned above, what about factors like luck and opportunities? How do you consider these issues? Anyway, this paper still gives us a lot of inspiration and encouragement. Thanks again!

[ruixili]

Thank you in advance for your presentation. The two papers - Early-Career Setback and Future Career Impact and Quantifying dynamics of failure across science, startups, and security are really interesting. I wonder how to validate the result. How did you construct the counterfactual of a person's career?

[boyangqu]

Thanks in advance for your presentation! This research is really encouraging in pointing out correlation between failure and opportunities in one's career path. My question is how can this relationship consistently be addressed with real-world situations or rationales? And how do you consider the role of early success in opposite hand?