10/17—James Evans

[smiklin]

Comment below with questions or thoughts about the reading for this week's workshop. See here for guidelines on what kind of content to post. Please make your comments by Wednesday 11:59 PM, and upvote at least five of your peers' comments on Thursday.

[Yilun0221]

Thanks for posting the exciting reading! As you mentioned in this article, many breakthroughs are often made by a research team that is far apart. I have read another of your papers, which said that the larger the research team, the closer the connections between large-scale teams, the easier it is to reduce the originality and subversiveness of their research results. In contrast, small-scale research teams have a relatively loose organizational structure, are at the edge of the research field, and cannot be closely linked to the core team in the research field. So my question is, how can we balance the relationship between teamwork and personal innovation in the research team?

[wanitchayap]

Dear Prof. Evans, I really enjoy this reading! In the paper, you only explore research in STEM. Do you think this model will fit well with research in social sciences without further modifications? I believe that the general framework of how you approach this research question regarding research in social sciences would be the same. However, I am also not sure if some details in methodology may needed to be modified. My reason is that novelty and related measures in social sciences might be harder to quantify. For example, patents as a measure is more relevant to STEM, and only social sciences category in nobel prize is Economics. And if this approach is applied with social science research, do you think the same results with the STEM would be expected? Especially regarding your demonstration that innovation in STEM is most surprising when certain research questions are answered by people from distant fields. I think that in social sciences, it is more often that different disciplines will ask the same research questions, but tackle those questions with different approaches or level of analysis. For example, both Psychology and Sociology research about mental health, but they employ slightly different views and tools.

[policyglot]

Mint (@wanitchayap) makes a great point about modifications for social sciences. Let's consider the recently announced Nobel Prizes. The only one for social sciences- in Economics- was awarded to Abhijit Banerjee, Esther Duflo and Michael Kremer. Their core innovation- spread across 70-80 experimental studies (Biswas, 2019)- involves an example of the interdisciplinary transgressions that you mentioned in your paper. They took the method of randomized controlled trials- hitherto used in the pharmaceutical industry- and applied them to development economics. However, we face two anomalies in their selection: 1) Their approach can no longer be considered novel. It has promoted vast legions of similar studies, including an entire online repository at 3IE. https://www.3ieimpact.org/evidence-hub/impact-evaluation-repository Furthermore, the areas of experimental economic research- both in terms of domain and geography- have expanded considerably. Would their work then be considered to have context and/or content novelty? 2) Their most cited paper (link below) involves surveys in developing countries, but makes no mention of Randomized Controlled Trials. How would your methodology deal with such cases, given that the most cited paper does not seem to be the award winning one? https://courses.cs.washington.edu/courses/csep590b/08sp/docs_p/banerjee_duflo.pdf

Thank you once again for this stimulating research. In the same vein, we hope to keep our eyes and ears open to how we can work across subjects for pathbreaking ideas.

References Biswas, Soutik, Oct 15, 2019, ‘Abhijit Banerjee and Esther Duflo: The Nobel couple fighting poverty’, BBC. Available at: https://www.bbc.com/news/world-asia-india-50048519

[tonofshell]

The status of research universities not only depends on the quantity of research generated, but also the presence of faculty and research that are deemed to produce scientific breakthroughs. Your paper proposes that "education seeking to cultivate scientific breakthroughs might teach trans-disciplinary search for problems". This suggests that schools might pursue growth in interdisciplinary programs (perhaps like our own), instead of programs in specific subject areas, as a way to better foster scientific innovation, and make the school more desirable to prospective students. What do you think the implications of this transition may be for the future of higher education?

[ShuyanHuang]

Thanks for presenting. I think it would be interesting if we consider the combinations of academic fields and the development of knowledge in a dynamic setting. For example, instead of considering a one-time combination, we may consider the evolution of such related combinations over time, and it’s relationship with the spread of some interdisciplinary methods.

[ydeng117]

Thanks for presenting. This paper definitely gives cogent evidence to convince scientists to think out of the box. In social science, however, scholars having various visions and ideologies could choose different paradigms. More specifically, in sociology, there are functionalism, interactionalism, Marxian class conflict theory, and etc. In economics, the orthodox neoclassical school is constantly challenged by Kneysian, institutional, Marxian, and Austrian thoughts. Will collaborations among social scientists with different schools of thought also stimulate innovation and predict breakthroughs?

[timqzhang]

Thank you for your article ! I am quite impressed by the idea since it tackles inventions from a new perspective. It measures the significance of breaking the boundaries between disciplines to boost the innovative ideas. Here I have two following questions.

1. It is important to settle the criteria of what disciplines should be included to obtain the “surprise” potential, which arises the following issue: the first one is how “distant” should two or several disciplinary sources defined, and categorized. In other words, what combinations could be considered as a combination with “surprise” potential?

2. As for the awarding mechanism, how to awarding institute identify those good combinations and bad ones, since they could not award all “surprising” combinations? It is also worth considering whether we should divide them into good and bad ones, or treat them equally, because it may be hard to foresee the outcome of those combination right now.

[nwrim]

Thank you for the great presentation! I have a question regarding the citation count. I think there are many factors that could affect the citation count but are not directly related to the significance of the paper. Two factors that I can think of now are (1) size of the field/subfield and (2) citing style of the field/subfield. If there are 100,000 researchers working on field A and only 1,000 researchers working on field B, then an article in field A is likely to get more citation than an article in field B. This leads me to think that size of the field that the article is related to will be a confounding factor.

Also, there are various fields in science and each field has their unique style of citing. These variability in citing style could affect the citation count. For example, it might be a convention in field A to cite the all the important article which contributed in figuring out a phenomenon, but it might be good enough to cite the most recent review article about a phenomenon in field B. Analogous to the size of the field, an article in field A will be likely to get more citation than that of field B.

As an indirect example, four out of five most cited article between 2014 and 2018 is all in the field of AI and machine learning (according to google scholar). The papers discussed in the presentation today are all from the field of biomedicine or physical science, so I think there will be little or no field variability. However, there are many subfields inside these fields. For example, particle physics and optics both reside in the realm of physics, but two fields are vastly different. Therefore, I think the factors I listed here will still play a role in the citation count.

How should we account for the confounding factors like this in using citation count? Do you think these factors are negligible? If not, how can we control for these factors?

[MegicLF]

Thanks for sharing such exciting research. It is really interesting to see the quantitative analysis of the novelty of the published papers, which is one of the most important characteristics of scientific research. In this article, you examine the novelty in two categories, content and context novelty. And we can tell from your work that the content novelty is more significant than the context novelty regards to the citation rate. But the curves of both increases as the citation quantile increases. I wonder, in the research, if you found any paper that has relatively large content novelty with below-average context novelty. How do you value the content novelty and context novelty in the publications regardless of the citation rate (since sometimes the topic is too new to be accepted)? Did you find any significant differences in novelty between each category of publications (biological science, physical science, and patents)?

[vinsonyz]

Thanks for your presenting, Dr. Evans. Since last century, with the development of the computer science and internet, the cost of obtaining data has been decreasing. So, there were many inter-discipline researches with a combination of social science, statistics and computer science. In addition, in the past decades, the preference of research has shifted from the theoretical analysis to empirical analysis. Therefore, my question is, the data and computer science have already changed our way of conducting researches, what could be the potential candidates of the novel idea or technology in the future that can be applied to the existing subjects?

[bjcliang-uchi]

Thank you for presenting this amazing research. Going off of the point raised by @wanitchayap, @nwrim and @policyglot in regards to how the level of novelty varies across different disciplines, I am thinking that novelty in applied science is perhaps different from that in pure theory. The the importance of the latter, for example, may take much longer time to be recognized for the general scientific community, especially when we consider the "established" schools of thought which more or less suppress certain groundbreaking ideas in each discipline.

I also have a clarification question: what exactly is the connection between "surprise" and "novelty"?

[yongfeilu]

Thanks for the great presentation, Dr. Evans! How can we explain these years' Nobel Prize awarding using your model frame? Would the academic institutions favor those who explain certain issues powerfully using simple and easy-to-approach method or those who create theories with normal explanatory power but using innovative and sophisticated methods? Is there possibly a pattern? By the way, could you please explain a bit more about context and content?

[YanjieZhou]

Thanks for the great presentation. There are many inter-disciplinary researches nowadays trying to find the new possiblities hided between the boundaries of disciplines. But as far as I am concerned, breakthroughs stemming from the combination of different areas require researchers to have a mastery of those areas, which seems pretty demanding compared with being familiar with only one area. For we graduate students, who still need improvement in many aspects, how can we balance between spending time on acquirement of inside knowledge and outside knowledge?

[KenChenCompEcon]

Thanks for the interesting presentation. It has become so luminant that a set of new researches have been greatly enhanced by introducing computational methods and machine learning tricks into their work. Social science, as well as other disciplines such as chemistry, biology are so hooked with making their research computational. Some of them are groundbreaking works but some are just playing with these new fashions. Do you think there is a pattern, or anything in common that all the successful academic combinations all share?

[HaowenShang]

Really interesting topic! After reading the paper, I am quite confused about Figure S3. It shows that when team novelty is low and career novelty is low, the joint impact of team novelty and career novelty is higher than when team novelty is low and career novelty is high. In other words, the lowest joint impact occurs when team novelty is low and career novelty is high. How to explain that phenomenon？ Thanks!

[JuneZzj]

Thank you for giving the presentation. As you mentioned, the findings about the unpredicted breakthrough in the field of science contradicts with the traditional understanding of how the collective attention works. The cross-disciplinary process tend to obtain more from a foreign findings. In terms of the content and context citation, the scientists tend to cite those are familiar to those venues than those are distant. While the investors tend to be indifferent between citing closer or distant. Does the difference in the behavior somehow influence the potential breakthrough can be realized among scientists and inventors. Does the tendency of the behavior have some negative impact to some extent? Thanks.

[dongchengecon]

Thanks a lot for the presentation! The idea of the paper is quite interesting and you have proved us the importance of interdisciplinary pattern of research. Based on your model and argument, I am wondering if that would be possible to create a "distance" measurement to quantify the linkage between each discipline? It could be roughly built on the connectivity of the contents, contexts in the papers. And we could also measure one subfield's overall "market access" to all the other fields.

[WMhYang]

Thank you very much for the presentation in advance. In the end of your paper, you mentioned that education seeking to cultivate scientific breakthroughs should encourage trans-disciplinary search for problem and combine students or teams from different backgrounds. I have two questions about this point.

1. Given that we cannot predict exactly which combination of subjects could lead to an outsized success, or put it in another way, the probability of a finite combination of backgrounds and subjects to succeed is extremely small, how could we justify the value of this kind of education if most of our experiments to combine different subjects and backgrounds fail? More specifically, how could we compensate the failed experiments?

2. When computational social science was a new combination, it really made a lot of breakthroughs in both social science and data science. But now it has become a commonly used method to tackle with social science and data science problems. For us MACSS students, if we are ambitious enough to make a outsized breakthrough, what do you recommend us do to gain new insights and to find meaningful combination?

[sunying2018]

Thanks for sharing this interesting research. This research demonstrates the motivations of breakthrough discoveries from an innovative perspective. This paper examines how novelty is associated with publication citation impact and awards by dividing MEDLINE papers into 10 equal-sized groups based on citation counts. As we know, citation counts may be influenced by a lot of factors.( i.e. For some specific area, there are much less researchers.) Could we also consider this problem and group these papers into different categories according to their specific area and then compare the citation counts in each category?

[ShanglunLi]

Thanks for providing such an interesting paper to read! I am wondering how is the successful academic breakthrough defined. The one who finds a simple method to solve a problem or the one who establishes new theories using complex models and ideas? Is there a pattern that we can investigate and make it clear? Thanks!

[SiyuanPengMike]

Thanks a lot for this interesting topic. After reading it, I'm still curious about the relationship between a successful breakthrough and a diverse background. Is there a way to clearly quantify the correlation between each subject and the breakthrough in a specific domain? Biomedical, physics, and chemistry are rather similar domains. Will a combination of irrelevant domains still bring up a similar result in your paper?

[linghui-wu]

Thank you for your presentation! The findings lend substantial support to the necessary combination of different disciplines and the existence of interdisciplinary educational programs such as MACSS. My question is that what is the acceptable "distance" between different subjects? For example, it is seemingly more natural to apply data science knowledge in social science to analyze issues at stake, or to employ computational techniques to assist biology; however, such a rule may be hard to be applied between natural science and humanities.

[zeyuxu1997]

Thanks for presentation. The paper show interesting and surprisingly accurate method to predict content and context combinations. Since you use data of a long period, I have a question that did you find some interesting structural change on the pattern of surprising findings derived from combinations of different area, for example, the percentage of those surprising findings from combinations among all surprising findings, or the average number of areas related to one finding?

[SoyBison]

Thanks for presenting your research! The findings seem to support the intuition that interdisciplinary study benefits both fields involved. That being said, a lot of scientists have trouble reaching outside their field to form these novel teams. I wonder if working on novel ideas experience long-term changes to their process. As a sort of expansion of that same idea, do you think that people who have worked in a novel team in the past are more likely to put forward novel ideas regardless of their team in the future?

[dhruvalb]

Thanks for sharing this work! Can you please explain the features of this model - "complex hypergraph... using mixed-membership, high-dimensional stochastic block models" and share the thought process on why you decided to model this phenomenon as such?

[chun-hu]

Thanks for sharing this exciting paper! I'm quite interested in your methodology to build the higher-order stochastic block model. Could you elaborate on how you quantify career novelty, team novelty, and expedition novelty in this model, as these concepts seem quite subjective and the definitions could be different depending on contexts?

[RuoyunTan]

Thank you for sharing your work with us. Could you share more thoughts on this drastic difference between science and technology in terms of the "distribution of collective attention"? Your research shows that scientists tend to collectively focus on and dig deeper into more concentrated and thus narrower topics, while inventors search across different fields as needed by their work. Could this mean a potential divergence between scientific and technical institutions in terms of their operation and progress, even though these two industries are closely connected? What opinions or suggestions do you have for these two kinds of institutions on this phenomenon?

[di-Tong]

Thanks for sharing this interesting paper! While the key finding of this paper is that scientific breakthroughs occur when problems from field A are unexpectedly solved by researchers from a distant other (field B), I wonder if there exists a sort of limitation on how distant/surprising is field B from/to field A to promote innovation. For instance, when scholars in the field of data science, complex systems and engineering write about the general social science topic, e.g. job polarization and income inequality, they tend to only borrow insights from economics among all social science disciplines, though there're tons of sociological works that produce very insightful theoretical frameworks and analyses. To explain this phenomena, my guess is that the frameworks provided by economics are more easily measurable and testable through modeling. Besides, the prevalent methodology utilized in economic literature might make more sense to scientist outside of the social sciences. In this case, there seems to be a boundary for interdisciplinary influence (the "surprising effect"). Is there a way to rigorously define and measure such boundary?

[keertanavc]

Really good research! This study proves how valuable interdisciplinary research is and rightly mentions in the limitations section that this could lead to negative crowd sourcing. How do you think this could make a case for journals to publish unsuccessful studies?

[lulululugagaga]

Thanks for your presentation. My question is that when interdisciplinary projects become the next potential breakthrough, it may run into new legal or ethical issues that are not defined by both disciplines, but who should be responsible for making boundaries of newly-developed research?

[ziwnchen]

Thanks for sharing! I have two questions:

1. This paper examines its prediction performance about next year's combination by comparing it with the hypergraph built on next year's ontologies. However, ontologies can actually change from year to year. In fact, one possible consequence of a big scientific breakthrough is that a new field is added to the ontologies (e.g, computation social science may not exist before 2009). However, the prediction model built on the previous year's ontology may fail to capture those new appearing nodes. So my question is, is it possible that the prediction model can only predict normal and self-similar combinations accurately and misses some of the biggest breakthroughs?

2. Although citation has a high correlation with content and context novelty, that does not mean that novelty lead to a hit paper, right?

[romanticmonkey]

Thank you for you presentation! Interdisciplinarity is indeed a key to new perspectives in scientific pursuit. However, I was wondering if individual differences apart from the number of fields a scientist has been trained in exists. For example, might the scientist be raised in a environment that promoted interdisciplinary creativity? Is there cognitive/developmental factors that boosts their performance in produce scientific breakthroughs?

[hesongrun]

Dear Professor Evans, thank you so much for your wonderful presentation. It is indeed inspiring and eye-opening. Can you offer an intuitive interpretation of neural network model you employed? How can we understand its nodes and hidden layers? Is there any survivorship bias in your study? Because it can be the case that many papers that draw bold connection between two distinctive fields may not get published in the first place. Only few gets published and achieves mass success. Thank you so much!

[anuraag94]

Thanks in advance for your presentation Professor Evans. I'm interested in your thoughts on the role of surprise in the line colloquially drawn between innovation and invention.

Let's loosely define the two terms as follows:

• Innovation: Discovery that leads to incremental progress
• Invention: Discovery that leads to a paradigmatic shift in the space

A paradigmatic shift is only visible in hindsight, after the new knowledge has been incorporated into the discipline as a subfield or folk knowledge. You remark on the scientific community/context undervaluing "breakthrough advances that transgress established boundaries."

However, if some true breakthroughs are never canonized within any discipline because of the conservatism you described, how does that square with the colloquial notion of invention as paradigmatic shift? How would you choose to model invention in this scenario?

[nt546]

Thanks for presenting such an interesting research work! I was wondering if you observed any surprising patterns for the impact of team and career novelty on the probability of a paper being hit over time?

[anqi-hu]

Thank you for the presentation! Regarding your findings on the relationship between a paper's unexpected novelty (esp. team novelty here) and its becoming a hit, how would you account for the fact that interdisciplinary papers, or papers with interdisciplinary authors, may naturally receive a greater amount of attention from multiple disciplines?

[bazirou]

Great presentation! Thanks for your share of new development on content analysis, and my question is about the generalization ability of the method and model in the paper. How do think the generalization ability of your method framework, can those surprising results be applied to other areas like social media, financial management and so on?

[rkcatipon]

Dear Professor Evans, thank you for the article! I look forward to hearing more about your work.

While the potential for interdiscplinary research to produce novel results is immense, is it possible that when fields "cross-pollinate" they can also replicate issues that plague their respective disciplines?

Furthermore, to piggyback off of @anuraag94's response, I would also like to discuss whether you believe the establishment of these field boundaries, was an intentional and negotiated process that was developed over time to help mitigate error in research.

[fulinguo]

Thanks for your interesting research. I am curious about whether there exist some micro foundations for the model in the paper. That is, could we model the researchers' decision making process based on their preferences and the corresponding utility function, and show how the model could help explain the patterns of breakthrough discoveries.

[yutianlai]

Thanks for the presentation. My question is how would the findings potentially impact industry. Interdisciplinary research is more recognized and frequent in academia, while in industry, companies' research labs might not have enough numbers of research teams to "collaborate" on one problem. What suggestion would you make under such circumstance?

[YuxinNg]

Thanks for the reading. It's impressive. In your article, you claim "Most of those unpredictable successes occurred not necessarily through interdisciplinary careers or multi-disciplinary teams, but from scientists in one domain solving problems in a distant other". The data analyses you run may support this claim, but I also want to know why and how this happens in real situations. I believe most scholars will tend to focus on their own research fields or the ones that are connected to their own research interests. Scientists may seldom try to solve a problem in a very distant domain deliberately. You mentioned Nobel Prize in your article, but as we can see almost every Nobel Prize is given to the researcher who is in this domain and made great contributions to this domain. So does that mean the surprise breakthroughs you mentioned in the article are literarily "surprise" or we can say just happened randomly or accidentally? Does it mean there is no way for us to generate these breakthroughs constantly? What we can do is just wait for the next "accident"? Thanks!

[liu431]

Thank you for the talk. You suggest "frame every student, team, and expedition as an experiment" in the last part of the paper. I understand this would help "radically alter science" from a broad perspective. However, for students or teams, how to motivate them to take the risk of failing when they face the pressure of graduation/tenure/funding?

[chiayunc]

Thank you for the presentation. Could you describe from the findings in the research and a social science point of view, what is the generative process behind the emergence of academic novelty? Is this more of a reflection of the process of how society has been generating knowledge, or surprises that are bound to happen with simply combining and recombining ideas? More specifically, does novelty here entails merely filling pre-existing holes that might not need filling? or a product of a generative process that pushes forward academic discussions?

[jtschoi]

Thank you for the wonderful article. As a student concentrating on computational economics, I am slowly realizing that inter- or trans-disciplinary approaches, as well as directly importing research methods popularized in other literature (including the use of RCT as @policyglot has mentioned previously), are increasingly becoming important in the field of economics.

One (theoretical) study that came to my mind (especially as this is a paper in my reading list) after reading yours was Noisy Talk by Blume, Board, and Kawamura. In this paper, the researchers try to analyze the context of obstacle in communication game also from a computer-scientific perspective (although not stated explicitly). That is, instead of simply looking at the strategic obstacles to communication as economists would do normally, they also analyze the technological obstacle to communication (or noise) and show how this technological obstacle may help reduce the strategic obstacle. This I think is a suitable example for "scientists in one domain solving problems in a distant other," as mentioned in your paper (and also as @YuxinNg has also quoted). Both computer scientists and economists are trying to crack the problem of efficient communication ("same domain" of problems), but from different points of view.

[ellenhsieh]

Thank you for presenting this interesting research! As you mentioned that "Most of these breakthroughs occur when problems from one field are unexpectedly solved by researchers from a distant other", I am wondering that what the word "distant" means here in this context. For example, does physics and chemistry distant from each other? or is it like , for example, computer science and social science are more "distant" from each other? Also, I am also curious about how you measure such as the novelty and breakthrough in this research, would you mind to elaborate it a little bit more?

[boyangqu]

Thanks for your presentation! This topic is very interesting and discusses a new field as far as I consider. While I am a question about its relativity in different time periods. While I understand this paper utilizes a large amount of data, I wonder that if the accuracy fluctuates in different eras, since popular topics change all the time (ex. AI machine learning as for now)?

[SixueLiu96]

Thanks for your presentation! I am really looking forward to it. You mentioned compound disciplines (or the interdisciplinary disciplines) will be the future of study. Based on this idea, after collecting tens of millions of research papers, patents, and researchers, your team construct a model correctly predicted more than 95% of next year's content. I must say this is very impressed. Therefore, my question is: How do you understand and predict this trend with more and more compound disciplines? Do you think that the future development of this trend is that everyone will learn more contents and gradually will be only a few disciplines?

[ChivLiu]

Thank you for the presentation! I guess this article just solved my recent confusion about how to justify the creation or invention of a paper. I have read through a lot of recent social science papers that were trying to use modern analytic tools to bring recurrences of experiments from years ago. Just in the summer, I heard from a sociology professor that he thought almost half of those reproductions were meaningless since they had just applied computational methods to prove some aged conclusions that people nowadays have considered outdated. Or, on the other hand, some researchers proved some famous theories would not be suitable for society at the present, but they stopped right there and started preparing to publish the paper instead of thinking of and revising the theories. I wonder how you think about the phenomenon. Would this bring positive effects to the development of computational social science?

[minminfly68]

Thanks for the working paper and interesting topic mentioned. Nowadays, interdisciplinary has become more and more popular in both academia and industry and jumping out of the box would be definitely beneficial towards the person (especially like me). However, my concern is that how far the connection it would be? I can fully understand the connection between computer science and social science, but what about philosophy and computer science or something like that? As you may find, not everything would be quantitative or computational, hence from traditional school's perspective, I quite doubt the outreach and boundary of the research connection among different fields. Thx!

[luyingjiang]

Thank you for your presentation. I am impressive with the precision of the model. I can see the point of unpredictable successes made by the interdisciplinary teams. But I do not quite understand the point that "but from scientists in one domain solving problems in a distant other". How would you define the extent of "distant“？Do you have any real life examples to support this statement? Thank you.