10/22: Berk Can Deniz

[ehuppert]

Comment below with questions or thoughts about the reading for this week's workshop.

Please make your comments by Wednesday 11:59 PM, and upvote at least five of your peers' comments on Thursday prior to the workshop. You need to use 'thumbs-up' for your reactions to count towards 'top comments,' but you can use other emojis on top of the thumbs up.

[LFShan]

Thank you for your presentation. I have never heard about A/B testing until I read your paper. I think it is a very innovative way to disincentivize innovation. However, I do think it could be a very reliable tool for industries that do not rely on heavy innovations. Could it be the case that for some of the industries it is the optimal way to increase profit, even in the long run?

[alevi98]

Hey, thanks for coming to our workshop and for the fascinating topic! I don't know a whole lot about A/B testing, but the focus on "incrementalism" makes me think of a similar idea in policy and government. I'm wondering if there's an analogue. Might we expect a more incremental growth process for the proposals of policymakers if them implement more A/B testing to see if people are in favor of their policies?

[chun-hu]

Thanks for coming! I'm wondering if the network effect plays a role in interpreting the result of A/B testing?

[Qiuyu-Li]

Thank you for sharing this interesting paper! I'm impressed by the way you identify incremental change and radical innovation. Unlike academic institutions, where we can measure innovation explicitly through the number of patents or published papers, the innovation in media is difficult to quantify. And I find your way of utilizing HTML and CSS very imaginative! However, my primary concern is also related to this. To how much content do you think you can generalize your research result? Since innovation in different industries happens in various ways. And one might argue that the incremental changes in the newspaper are not comparable to, say, the incremental changes in the biomedical industry.

[NaiyuJ]

Thanks for sharing! That's a fascinating paper! I like how you step by step explain the logic flow behind this idea, how you clearly present the dependent variable and independent variable, and how you structure the paper in a way I can easily catch up. The empirical analysis part is interesting but only focus on the media industry. I actually want to know more about how the different natures of industry influence your conclusions.

[cytwill]

Thanks for your presentation. This is a really interesting study for innovation. As you find that "the adoption of A/B testing decreases the likelihood of radical change and makes websites more likely to change incrementally", have considered any more in-depth reasons for why these websites apply A/B tests or not? What are the differences between the two patterns in terms of benefits to the websites and how this might be related to the changes, or the "innovation"?

[goldengua]

It is really interesting to design an experiment with A/B testing. I find your result 'wesites are more likely to make incremental changes with A/B testing' very interesting because it is consistent with people's underlying psychological motivation to like familiar items. Do you think AB testing can be extended to psychology experiments to find out the motivation for incremental innovation?

[yongfeilu]

Thank you for your presentation. It's really exciting to see how you apply A/B testing to analyze web design. To which degree do you think your methods can capture the trend and features? What do you think your research can have any impact on web design? Could you please give me some perspectives? Thank you very much!

[MegicLF]

Thanks for bringing such an interesting topic! Your work reminded me of a discussion of why people use p=0.05 in my undergraduate statistic class. People use p=0.05 sometimes just because of conventions and it could make their research look rigorous. Since you are focusing on the newspaper industry, would you estimate a similar result on academic papers (especially in social sciences)? I am interested in your opinions on A/B testing used inside scientific research.

[jsoll1]

Thank you for sharing your paper with us! I know that you said your paper doesn't focus at all on how to preserve innovation while not abandoning A/B testing, but I'm still curious as to your thoughts on it! Do you think that increased decentralization in a company (maybe a little less than the uber-competitive environment of Sears!) could help reduce the first mechanism of primacy? And what kinds of office cultures do you think would help reduce the impact of the second mechanism you postulated?

[shenyc16]

Thank you for sharing this interesting research with us! I think using HTML and CSS to identify whether the change is incremental or radical is surprising and effective. However, I am still curious about the border of “innovation”. For example, if two versions of a newspaper website have little change in algorithm, like some small details in UI, but can significantly affect readers’ feeling, will this be counted as radical change? If so, how will you include this in your model?

[luxin-tian]

Thanks for sharing. It seems that A/B testing has been the mainstream methods for industrial firms perform hypothesis testing in business operations, and it is interesting to see you examined the outcome of this from an external perspective. Sometimes A/B sampling, though total randomization is pursued, can result in spill-over effects especially in a setting where individuals have interactions with each other, and the control groups and the treatment groups can be involved in endogeneity problem caused by the so-called simultaneous equation bias. How would we address this issue and how does it may affect the assumption of your research?

[Yaweili19]

Thank you for sharing your research! This topic seems very interesting. I never thought that the application of experimental methods could have such a significant impact on organization management outside of the academic field. Can you talk a bit in detail about how newspaper data support your research? I am not very clear on this point. Looking forward to your speech!

[ChivLiu]

Thank you for this wonderful presentation! I think that A/B test is very straightforward to get the testing results, but I wonder that would changing the sequence of the questions change the results of the testing results? I have seen many behavior tests using different orders of questions to test if the person was lying, but how could we verify that the results are accurate?

[afchao]

Despite some of my classmates' misgivings, I find it intuitively compelling that the adoption of A/B testing can lead to an organizational preference for incremental change in the context of business. As you wrote, the critical aspects of A/B testing that drive this are its speed and affordability, which reduce the costs of testing incremental changes to near-zero and, in turn, make any amount of benefit to be gained from such incremental changes highly profitable relative to their price.

My question is whether or not you think there is a limit to the amount of benefit that is possible to accumulate with incremental variations on a basic strategy. If so, shouldn't reaching this limit naturally force firms and agents to begin exploring more radical approaches to selection and evaluation? If not, why not?

To expand, and borrowing from your description of Knudsen and Levinthal's work, let's imagine a given firm "locked" on a hill in the business fitness landscape, where our firm only explores territory on the hill and boundaries of that hill delimit the extent to which any incremental change is capable of changing the basic nature of the business practices represented by that hill's position in the broader landscape. If the height of this hill corresponds to some metric of our firm's success, there should be one point on that hill which is highest (or tied for highest) and which represents the most successful possible variation on the overall theme of the strategies described by this hill. Once the firm stumbles onto this point, and given sufficient time to realize that they are indeed at the pinnacle of their current practices' potential, should we not expect the firm to then begin exploring methods of improving their success that might get them onto a different hill with higher still pinnacles?

Thank you for your time!

[jinfei1125]

Very interesting topic because nowadays Random Control Trial and A/B tests become so common and people respect them as golden rules. This article points out that sometimes experiments and A/B tests can damage innovation and creativity. My question will be, in the selection stage, do you think there is any selection bias? Thank you!

[xxicheng]

Thanks for sharing! I am really excited about tomorrow's presentation. Maybe a bit broad, but my question is, what do you think of the future application of A/B testing? Thanks.

[mingtao-gao]

Thank you for your presentation in advance! A/B testing is indeed a useful experiment using randomized controlled trails to test out alternatives and its design decides such an experiment is used mainly on testing out small changes made to the current design. My question is that if the results with A/B testing are enough to imply that experimentations can harm innovations given there are many different experimental methods targeting different aspects of a product/website/service?

[tianyueniu]

Thank you for sharing the paper with us! This research definitely makes me think of the 'The Social Dilemma', the Netflix documentary that talks about how tech companies are using A/B testing and other methods to subtly capture users' attention and cultivate addiction. Apart from potentially blocking radical innovation, do you think the wide adoption of A/B testing might lead to ethical violations as well?

[Jasmine97Huang]

Thank you in advance for presenting such interesting topic and perspectives! You made a deliberate choice to exclude changes in content when measuring changes on the U.S. newspaper websites. You justified this by arguing that "newspaper tend to experiment on designs not content." How did you come to the conclusion?

[Panyw97]

Thank you for sharing! It's really exciting to see A/B testing under a rigorous academic discussion. It's a vital analysis approach in industry sectors as well. Like many others mentioned previously, what is the underlying causality between the incremental action and the adoption of the A/B experiment?

[FranciscoRMendes]

Interesting, I think that what is implied by the paper, actually happens, because smaller companies are more aggressive in revamping their websites (basically anything), this is perhaps an unintended consequence of the small sample problem inherent in A/B testing.

I am also interested in thinking about whether "small" changes in the CSS and HTML script are significant? How do you statistically deal with this problem?

[97seshu]

Very interesting paper! I am used to think that A/B testing promotes innovation because it is mainly used for improving websites and applications. This paper provides me a new angle to look at it. Do you think in what situations are A/B testing a more suitable method to use? And in what situations, it will be better to use other methods?

[ttsujikawa]

Very interesting insights into evaluating the experiment method. Through your research, I could easily imagine that the conventional A/B method possibly discourages people to be innovative in the context of the website. Despite your amazing idea, I think that this research is only applicable to the context of the website, and it is hard to see innovative changes in website designs. Thus, as further research, I would like to see that you investigate that this phenomenon occurs beyond the field of website design. Also, as a modest comment, I agree with the statement in which A/B testing is to assess innovative ideas. In other words, it is not generally utilized to generate innovative ideas.

[boyafu]

Thanks for sharing! Since the A/B testing is becoming more and more popular across companies, some may know little about the testing itself and merely consider it as a "prerequisite" for changes. I was wondering if having a better understanding of the A/B testing itself or not may play a role in the process of innovations.

[XinSu6]

Thank you so much for the paper. Its very insightful. Never thought about A/B testing can be applied in this way. Like many others have asked already, I am wondering what is the justification for the existence of the causality between the incremental action and the A/B testing experiment. Thank you!

[JuneZzj]

Thanks for presenting. A/B testing has been widely used in industries and it is astonishing to know that it actually slows the innovation process. Once firms realize how A/B testing could change their product management, what the next step do you think firms may adapt to replace this methodology? Thanks.

[chrismaurice0]

Looking forward to discussing tomorrow! There has recently been much consolidation of the newspaper market in the United States, as a result of consumers prefer subscribing to a few national papers and neglecting their local papers. Do you think A/B testing can be used by smaller local/regional papers to combat the consolidation of the market? Or does A/B testing solidify the creation of an oligopoly between the New York Times, Wall Street Journal, and The Washington Post(with potentially the inclusion of the Boston Globe, LA Times, and Chicago Tribue) given smaller papers have to rely on ad revenue, which takes away from the experience of reading the news (for example, when reading an article on the Chicago Suntimes website, ads fill nearly the entire page)? 

[chuqingzhao]

Thank you for sharing for your work! It is a very interesting paper and I enjoy the idea of analyzing A/B impact by using newspaper data. Here is my question : 1. As this paper focuses on the newspaper data, I am wondering what is the impact of A/B testing on other industries, such as e-commerce platforms? 2. By adopting A/B testing, new ideas are less likely understood. I am wondering what is the next step of A/B testing? Or how to make incremental change as well as understand new ideas in industry. Thank you!

[harryx113]

Thanks for the presentation. What is your take on the impact of A/B testing on user addiction? When content remains the same, A/B testing aims to maximize user engagement and retention, which would generate more ad revenue for the companies. How do you view the future of A/B testing in this context?

[ghost]

Do you think the popularity of A/B testing will ever decrease?

[adarshmathew]

Thank you for this paper! It confirms quite neatly my pre-existing distaste for A/B testing-driven development.

My question is tangential to your paper though: Can your conclusions about incrementalism be extended to RCTs, the A/B test's cousin (uncle?)? They've been lauded for their 'evidence-based' approaches, but your critique here should apply to them too, right? As such, is the development world's fascination with RCTs just academic incrementalism?

[YaoYao121]

Thanks for the paper and the presentation! It is really interesting and fantastic. Well, I am very curious about how could the findings about A/B testing here could effect concerning field in academic research and applied business.

[k-partha]

Thanks for taking the effort to present your paper. To what extent do you think we can make causal claims given this data? Do you think this relationship would hold in other industries?

[weijiexu-charlie]

Thanks for your presentation. I'm wondering how to extend your conclusion to other fields/industries?

[rkcatipon]

Thank you for sharing your reseach with us! I too have very mixed feelings about the use of A/B testing for newspapers. I think that it encourages the optimization of headlines for clicks and does not prioritize an informed public. And while your paper does not look at the change in content through A/B testing, I wondered if the prevalence of A/B testing and the shifting of newspapers to news websites as core products, is also reflective of a growing loss of revenue that journalistic outlets faced in the digital transformation of news. Are industries more likely to embrace incrementalism when faced with medium obsolescence?

[jkatz913]

What are (some of) the incentives driving managers and others in position making decisions to err on the side of caution and incrementalism?

[Rui-echo-Pan]

Thanks for sharing! Could you share more about how experimental and behavioral methods can be used in studies in the economics/business field?

[aolajide]

I wonder if you, Berk Can Deniz, have speculated about any solution for the problem of incremental change vs. radical innovation you mentioned in your paper. For example, instead of A/B testing would A testing work where the A test is really meaningful pointed in the direction of creative change?

[Yiqing-Zh]

Thank you for your presentation! I am wondering whether the effect of A/B testing can be applied to other fields and whether it is a causal relationship between experimentation and innovation.

[luckycindyyx]

Thank you so much for sharing! I also have questions about A/B test. According to the main potential concern of A/B test in rigorous academic research, what is the future trend of improvement/replacement? Also I noticed that you think newspaper experiment on the designs instead of content. Could you expand on the origin of this point of view? Thanks!

[ziwnchen]

Thanks for sharing this interesting idea! It is really inspiring to see that the choice of evaluation methods (e.g., A/B testing) could essentially shape the process of organizational innovation. My question is related to your choice of the dataset when exploring how to experiment influence innovation.

For websites, is it always the case that large transformation equals to innovation while small changes lead to incrementalism? Is it possible that some small changes could really make a difference? That said, how could you evaluate the novelty level of website changes?

Also, is it possible that there are some potential confounding variables? For example, large newspapers with a longer history tend to be both more cautious in adopting large changes and more willing to use the A/B test (because it is more standardized)?

[WMhYang]

Thank you very much for your presentation. It is a great paper trying to investigate the relationship between experimentation and innovation through the lens of A/B testing. However, I am not quite familiar with the context, so could you please give me some other detailed applications in social science researches?

[caibengbu]

Thanks so much for sharing the work!

I am very interested in hearing more about how A/B testing could potentially contribute to other realms beyond media and business and whether the effect of A/B testing can be applied to other fields and whether it is a causal relationship between experimentation and innovation.

[timqzhang]

Thank you for your paper ! I have a general question beyond the A/B test itself, like how to choose the proper statistical tests given the properties/features of data? I mean different methods may lead to totally different result, statistically.

[zixu12]

Thanks so much for sharing work. As the A/B testing is new to me, I am wondering how would it be applied to fields such as development economics. Intuitively I think it should be useful in this field. Also, is there ant bias you think we should be specially paid attention to when dealing with A/B test? Thanks!

[fyzh-git]

Thank you for this interesting paper! The novel method of measuring the evolution of websites by comparing the changes in their source codes is enlightening, and the combination of the history archives with the records of whether or not adopting the A/B testing truly contribute to the past data limitations. But I'm also wondering whether the results and conclusion could apply to the non-established companies as well? Thanks!

[Tanzi11]

I am looking forward to your presentation! In regard to the media industry, how has A/B testing helped (if at all) the current issues they are facing such as local journalism becoming defunct or bringing in alternative sources of revenue (outside of advertising)? Thanks!

[romanticmonkey]

Thank you for presenting this study to us! As I'm reading your paper, I can't stop thinking if incremental change in websites and apps that involves user's content creation would better stimulate user's creativity than radical changes. For example, would incremental changes in YouTube encourages better content quality and quantity than radical changes? Like subtle changes in upload features?

[YileC928]

Thank you so much for bringing us such an intriguing paper! and I really love the way to present the paper (clear and engaging)! Your novel finding of the impediment effect of A/B testing on innovation and the computational methodology adopted really open a lot of opportunities for future researches. Could you maybe expand a little more on that? For instance, how might the possible mechanisms proposed in the paper be operationalized and tested? Are you considering research further on the topics you brought up in the discussion section?