7. Accounting for Context - oritenting

[JunsolKim]

Post questions here for this week's oritenting readings: P. Vicinanza, A. Goldberg, S. Srivastava. 2020. “Quantifying Vision through Language Demonstrates that Visionary Ideas Come From the Periphery”.

[pranathiiyer]

I would like to know more about the granular aspects of how existing models can be fine tuned on new corpora of text? Throughout this quarter, we have encountered several papers that use various methods such as embeddings, language models, and topic modelling to identify and quantify certain constructs. How can we as students understand the suitability of different methods to achieve similar outcomes?

[isaduan]

I love the paper! But the authors' validation of the model seem to be cursory: they manually analyse what their model identifies as most and least visionary in each of the three settings. What are some other ways to validate that the measure of visions is a good measure?

[konratp]

I find it interesting that the authors rely on Google, a private company, to provide pre-trained models as training BERT from scratch is too resource-intensive. Are there concerns or potential conflicts around academic freedom if researchers rely on Google in applying their research methods? Would there be other ways to access pre-trained models that don't involve megacorporations like Google, which might have a vested interest in skewing skewing the pre-training of these models? Or am I just paranoid and is this actually just.. fine?

[Qiuyu-Li]

I was not quite persuaded by this paper... If I understand correctly, this paper first identifies visions by finding text features that had been innovative at a moment, then become commonplace. If this is true, the algorithm was not designed to perform prediction tasks, and it could at best find the novel ideas. Furthermore, I personally doubt what kind of novelty it found. Is it a novelty in idea or is it a novelty in word usage or sentence structure? Or perhaps the algorithm was just finding unusual ideas that happened to become visionary.

[chuqingzhao]

It is an amazing paper! I love the idea of measuring visionary idea through BERT model and how it impacts the firm's future success. One question for this paper is whether it is possible to take different contexts into consideration? Visionary idea can be dependent on contexts; for example, as the paper analyzed the public listed company, if we consider the startup context, the "visionary" idea might not be visionary any more in start-up contexts. I left wondering is it possible to answer where do visionary idea come from from several different contexts? If yes, whether we can detect some patterns of inter-organizational social learning?

[ValAlvernUChic]

I was not quite persuaded by this paper... If I understand correctly, this paper first identifies visions by finding text features that had been innovative at a moment, then become commonplace. If this is true, the algorithm was not designed to perform prediction tasks, and it could at best find the novel ideas. Furthermore, I personally doubt what kind of novelty it found. Is it a novelty in idea or is it a novelty in word usage or sentence structure? Or perhaps the algorithm was just finding unusual ideas that happened to become visionary.

Hello, Qiuyu! I think the authors clarify that visionary goes beyond just textual novelty but also how commensurate the ideas become from its inception to the future - if it goes from something previously unthinkable to contemporaneously attached to conventional logic, then it's visionary. They split its definition more finely into 1) rethinking the field's assumptions and 2) are basically prescient of how the field evolves. I quite like the first assumption, though the second assumption feels a bit like confirmation bias but I might be thinking about it wrongly. The context is important for their argument and can be done because of how cool BERT is!

On that note, it's pretty cool how they measure contextual novelty by comparing perplexity scores between two time periods. I'd be interested to see though how exactly the ideas have assimilated in the public consciousness! It is one thing to see how the ideas were visionary within the domains and another to see how it has also affected the ordinary citizen's understanding of the idea.

[Sirius2713]

This paper is amazing! I have a question about BERT: According to the description in this paper,

BERT instead favors bidirectionality—that is, it attends to both the left and right contexts simultaneously.

How does BERT differ from bidirectional Hidden Markov Model or LSTM?

[ValAlvernUChic]

This paper is amazing! I have a question about BERT: According to the description in this paper,

BERT instead favors bidirectionality—that is, it attends to both the left and right contexts simultaneously.

How does BERT differ from bidirectional Hidden Markov Model or LSTM?

BERT is the overall model built via a Transformer architecture - LSTMs are not models per se, they're a different architecture as well. The main difference is that transformers facilitate simultaneous learning of a sentence while LSTMs are sequential which can cause some long-term dependency issues if your sentence is super long - they can "forget". LSTMs have to go left-to-right then right-to-left, word to word thus not being simultaneous. So besides being faster, transformers more reliably "remember" sentences.

[facundosuenzo]

I was not quite persuaded by this paper... If I understand correctly, this paper first identifies visions by finding text features that had been innovative at a moment, then become commonplace. If this is true, the algorithm was not designed to perform prediction tasks, and it could at best find the novel ideas. Furthermore, I personally doubt what kind of novelty it found. Is it a novelty in idea or is it a novelty in word usage or sentence structure? Or perhaps the algorithm was just finding unusual ideas that happened to become visionary.

I'm also in two minds with the paper. While the idea of visionary it's interesting, I'm pretty confused by how they operationalized center/periphery. At the same time, as you mentioned, the findings look somehow expected (based on the sociology of organization literature and how innovation usually takes place there). Finally, I was trying to think of possible confounders of this relationship like individual traits or socio-demographics (socioeconomic status, education, and, mostly, gender). Ultimately, these other variables could change the "center" of our network.

[Jasmine97Huang]

The authors didn't mention the sizes of tokens across different time slices and how that might influence the contextual novelty that they are trying to capture. Would love to hear comments on that.

[mikepackard415]

This paper is interesting. As mentioned above I do think the "center/periphery" distinction deserves a little more explanation. In different domains they are operationalizing this concept different ways. In politics is has to do with network centrality in a cosponsorship network, in law it has to do with the level of the appeals court, in business it has to do with firm size.

[Hongkai040]

The authors did an interesting job. I think the definition of contextual novelty is a lot like the ‘novelty’ from week3’s reading “Individuals, institutions, and innovation in the debates of the French Revolution ”, where researchers also managed to discover the flows of ideas using topic modeling. So what’s the difference between these approaches and what are their advantages and disadvantages, respectively?

[hshi420]

I'm curious about novelty in different contexts. I think innovation can be different for different groups. It might be novelty for one group, but not for the other. How could we quantify relative contextual novelty that is embedded in different contexts that consider the novelties differently?

[Jiayu-Kang]

I would like to know more about the granular aspects of how existing models can be fine tuned on new corpora of text? Throughout this quarter, we have encountered several papers that use various methods such as embeddings, language models, and topic modelling to identify and quantify certain constructs. How can we as students understand the suitability of different methods to achieve similar outcomes?

^^ I'm also wondering how we should choose methods and fine-tune models to better fit our specific research questions, especially as students who do not have expertise on all available techniques.

[GabeNicholson]

Researchers often use the BERT base model as a starting point and then build off of that with their specific question. Out of curiosity, how hard is it to train one from scratch? I'm assuming it would take many hours and computation time, but how much exactly?

[NaiyuJ]

I'm kind of curious about in what situation it's better to use a pre-trained model and in what situation it's better to train the model by ourselves. It seems that training the model by ourselves would usually get better results if this is possible?

[LuZhang0128]

When reading the validation part, I'm a little concerned if the model really works or just the author telling a good story. I wonder if there's a more systematic way to validate the model? If I'm about to apply the model on my own dataset, it's probably difficult for me to tune the model if the validation is based on observation.

[YileC928]

The authors operationalize vision as the percentage decrease in 'contextual novelty' of a sentence between two time periods. I am left wondering how they partition the data and how they make sure that only comparing two periods (rather than a time series of periods) would be enough to identify 'novelty'. Could the patterns they discovered just by accident? Could it just be data mining?

[sizhenf]

I wonder if the author's definition of novelty is a standard in literature or there may be other ways to define it?

[kelseywu99]

It is a very interesting paper bringing up new questions and concerns regarding DL models. I was wondering the application of BERT model in the paper and why did the researchers specifically choose this model among others? What makes it stand out?

[sudhamshow]

I was wondering if the selection of source for training the BERT models could install any bias. It could be present in 2 ways - Leakage - If an idea is considered heretic to its time (theories of Galileo, Einstein) much of it may be 'news' and people start discussing about it, and might slip into literature. When the context is not realised when being trained the idea might not be recognised as novel although the context it appeared in was due to the idea's anomaly. Changing trends - An idea or concept might go out of trend and might later again be revived as novel when calculated in the way the paper does ( Vision as a difference of contextual novelty of different times). For e.g - the idea of neural networks were previously experimented by Marvin Minsky and Rosenblatt (Perceptron) and the interest grew around the subject. Because they weren't able to solve the XOR problem, the interest surrounding N.nets fell. The interest (Contextual novelty) has risen again with the introduction of multiple layers - albeit not a completely novel idea. But the authors' method might misrepresent it as novel.

I was also curious about the association of the research with the Principle of Abduction - while both have the same essence abduction relies on finding surprising evidence, while the paper's method does the opposite.

[ZacharyHinds]

When reading the validation part, I'm a little concerned if the model really works or just the author telling a good story. I wonder if there's a more systematic way to validate the model? If I'm about to apply the model on my own dataset, it's probably difficult for me to tune the model if the validation is based on observation.

I agree with this concern. In general, I find the concept of both quantifying and deriving "vision" to be hard to buy into. I would really like to see an article re-attempt these methods on other data and see if this is reproducible and generalization outside of the context they were tuning for. Generally though, what approaches are there to adequately validate these sorts of models?

[chentian418]

This is a an interesting paper to detect visionary ideas using BERT models. I have two questions: i) How good are two ingredients of vision—contextual novelty and prescience captures the true vision, since I am worried about the measurements detecting some constructed variables of vision rather the true visionary idea variable. ii) Is there any possibilities to visions included in video and audio as well?

[Emily-fyeh]

I like the discovery of this paper, and its interpretation of 'periphery' novelties in political, law, and business contexts. I would want to know how the measurement of different kinds of centralities can actually mean ground-breaking new ideas in a certain realm? For example, how do we interpret the difference of the eigenvector centralities and degree centralities in political data?

[ttsujikawa]

Great paper! I am not clear about the notion of contextual novelty that the authors attempt to capture. In the paper, they define it as 'the product of word-level perplexities in a sentence. What does that really mean in the real world context?

[melody1126]

The data used to measure vision in the business sector was based on publicly traded firms and the Q&A section of their quarterly earnings calls. (pages 10-11) However, visionary ideas often come from smaller firms that are not yet publicly traded, so this sampling error might affect the results and efficacy of the vision measurement from this paper. Moreover, it is also possible that the Q&A section of the quarterly earnings calls is not the most effective corpus because visionary ideas likely come from within-company communications on a smaller scale, rather than on these public documents.