10/28: Abdullah Almaatouq

[ehuppert]

Comment below with questions or thoughts about the reading for this week's workshop.

Please make your comments by Wednesday 11:59 PM, and upvote at least five of your peers' comments on Thursday prior to the workshop. You need to use 'thumbs-up' for your reactions to count towards 'top comments,' but you can use other emojis on top of the thumbs up.

[yjhuang99]

Hi Professor Almaatouq, thanks for coming to the workshop and presenting your working paper! Your work on the efficiency of interactive groups compared with individuals in different tasks of varying complexity is fascinating. There is an ancient proverb in Chinese that "Two heads are always better than one" (三个臭皮匠胜于诸葛亮), and it turns out the findings of your research provides more insight, especially that interacting groups act as fast as the fastest individual and more efficient than the most efficient individuals only in COMPLEX tasks so that this proverb does not hold in every situation. I would like to learn more about how you and your coauthors separate the effect of an individual and the interaction on the outcome in an interacting group (for example, this group solves a task efficiently not because of the interaction but due to a single talented person while others are just free-riding) and how to define the tasks as of similar difficulty level to individuals and groups.

[Thiyaghessan]

Hi Professor,

Thank you for providing us with access to both of your papers. I had three questions/points pertaining to avenues for future work in your research on task complexity.

1. You mentioned that a central puzzle that emerged was that groups, despite being faster and more efficient, do not find better solutions than the highest scoring individuals. I was wondering if that difference could be a function of time? In the short run, newly formed groups may not be able to produce the best solutions but in the long run over continued interactions, synergies could strengthen and result in improvements to the quality of solutions. Teams that work well together tend to have teammates who push one another to do better. While this may not be appropriate for lab experiments, perhaps field experiments looking at group performance in the long-run could help us understand if such effects exist?

2. I wonder if you considered examining the heterogeneity amongst groups. I understand that the paper compared groups to autonomous individuals but I was thinking that it might be interesting to look at variations in performance within groups. In practical experiences with teamwork, it is clear that some teams work better together than others. In this exercise, team members interacted with one another via text. Perhaps, similarities in communication styles between members of a group influence how well they work together with one another. A text analysis of the group chats could work here.

3. Group leaders. You talked about how groups take time to achieve consensus and thus having a group leader who can unilaterally decide if/when a task is complete or a solution good enough could help reduce the source of the delay. Once again, I wonder about the heterogeneity in leadership styles that could potentially cause variation in this variable. I doubt that all group leaders are equal thus a similar study where a randomly selected individual in each group is chosen to be a leader can help determine the types of leaders who actually do reduce delay. I am sure that poor team leaders can actually have the opposite effect and end up decreasing the efficiency/quality of solutions.

[mikepackard415]

Hi Professor Almaatouq, Thank you for sharing your work with us. After reading the paper about using high-throughput experimental designs in behavioral and social science, I had a question about the timescale of inquiry relative to change in the behavioral landscape. You mention that a high-throughput approach has been useful in sequencing the human genome, but as I understand it, the human genome is changing more slowly that human culture. My point is that, even if you can map a significantly large portion of the behavioral space across many dimensions, approaching general theories, isn't the ground truth of the behavior space shifting? I would think this would be especially true in today's age of mega-cities operating as social reactors and impending climate concerns promising to force behavioral change in the near future. Perhaps this makes the argument for high-throughput design stronger. Regardless, it seems important to recognize the difference in the stability of the problem.

I haven't gotten to your paper on task complexity and group synergy yet, but I'll be curious to see if your work draws on any of the work by Elinor Ostrom. In particular, David Sloan Wilson wrote in his book This View of Life that Ostrom's 8 principles for governing a commons were applicable to the management of groups.

[pranathiiyer]

Thanks so much for sharing your works Prof Almaatouq! They were an interesting read! I had two questions, more like doubts, based on your paper on throughput:

1. Generalisability- As i understand it to be, the experimental sample space starts out with a fairly general problem instead of a specific hypothesis and then collects information on all possible variables to study the problem. However, isn't the generalisability of the problem itself dependent on the mechanism chosen to operationalise it? For instance, the space of risky decisions could be studied by looking at gambling, or investment behaviour, or some kind of crowdsourcing maybe. In that sense will the space not always be constrained by how the construct itself is defined and measured? Or would the space for a different operationalisation be completely different?

2. This was just a thought but how do you think dimensionality of throughput experiments might materialise for experimental spaces where the data could be a heterogeneous combination of audio, video, images, text etc? Since the cost of processing these at large scale might be significantly different.

[hhx2207061197]

Thanks so much for sharing your works Prof Almaatouq! Looking forward to your presentation!

[jinfei1125]

Hi Prof. Almaatouq, thanks for coming to share your thoughts at our workshop! I love the famous quote you mentioned at the beginning of the paper "you cannot play 20 questions with nature and win." In your paper, you argue that given recent innovations and development, high-throughput approach is economically and technically feasible and would generate more reliable, more cumulative empirical and theoretical knowledge than the current paradigm and can do so far more efficiently. While this approach is efficient in social and behavioral sciences, can you elaborate more about those disciplinaries that would have to be restricted with current paradigm? Thank you!

[YLHan97]

Hi Professor Almaatouq, Thank you so much for sharing such interesting topic! I strongly support your argument that one of the keys to solving the puzzle is to better understand the underlying nature of the tasks being performed. The viewpoints in the article also brought me many new inspirations.

[fiofiofiona]

Hi Prof. Almaatouq, thank you for sharing your research at our workshop, they were interesting to read and also brought different perspectives to social and behavioral science research. Along with some of my peers, I am curious about the generalizability of the high-throughput experimental designs in different disciplines of social and behavior science; I also wonder what are the qualifications for an area to be "mature" enough that researchers can be confident using high-throughput experimentation results?

[yiq029]

Thanks so much for sharing your works Prof Almaatouq! Looking forward to your presentation!

[nijingwen]

Hi professor Almaatouq. Your study field : computer science & human behavior is really interesting and useful which can make big data used much better in social science. I am ready to hearing from you on Thursday!

[taizeyu]

Dear Prof. Almaatouq. Thank you very much for giving us a good speech. I am wondering that how can you combine the computational method into decision making

[borlasekn]

Hi Professor Almaatouq. Thank you for sharing your expertise with us! I was reviewing your paper on how task complexity impacts group synergy, and I was wondering if you had any insight to whether similar results would be true across age groups. This work reminded me of a show on Netflix, called "100 Humans" (a show of more playful experiments on these 100 humans). In one task, individuals were divided into groups based on age and were observed based on how well they completed a task. Do you think that, at different points in ones life, completing work as a group would be more beneficial? In particular, I am thinking about how we teach children scientific inquiry and problem solving and whether group work might be better or worse?

[YaoYao121]

Hi Prof. Almaatouq, thank you very much for sharing your research with us. This is quite an interesting research question about human behavior. I like your combination of computational methods and social science insights to study classical human behavior in detail. Your new experiment model, high throuhgput experimentation, really contributes on the weaknesses of traditional lab experiment. Even though you put some examples, I am very curious when following researchers apply your experimentation method to design new experiment-based behavior study, If there are some basic principles they need to obey? If there are, what are the principles, please? Thanks!

[yujing-syj]

Hi Prof. Almaatouq, thanks so much for discussing the paper with us. Looking forward to your interesting speech!

[xin2006]

Hi Prof. Almaatouq! Thanks so much for sharing the paper about experiment in social science with us. Coincidently, we learned different kinds of experiment, and corresponding pros and cons in this week. So I’m looking forward to hearing from you the “high-throughput” experiment and the way you run an experiment!

[afchao]

Thank you for sharing your work with our group! I regret to voice a general dissatisfaction about the basic premise of high-throughput experimentation: if it was so feasible, it would already be underway. Put more generously, I think the current structure of experimental social science is a reflection of how the theoretical basis of high-throughput experimentation intersects with real-world constraints. For example, a proposal is made in the paper for "an algorithm to procedurally generate the set of conditions in the experiment" - this is substantively different, on my interpretation, from an algorithm which could procedurally generate variations on the experiment itself, which seems like a better approach for exploring the experimental design space; in the former case your exploration is confined to the boundaries of that experiment's condition space. It seems clear that a truly comprehensive design space is a reflection of the theories which underlie a given experiment of the class "investigates this theory"; in order for your experiments to cover more of the design space they'd require more meaningful coverage of the underlying theoretical space, which in turn would require similarly meaningful variation in experimental method. Without getting too far into the obvious complexities in implementation on both the weak and strong formulations of this "procedural experiment generation" proposal, it seems clear that the amount of effort required for any such mechanism to actually exist is significantly lower than the amount of effort required to design a single experiment. Even making use of the in-development tools mentioned in the paper does not do away with this requirement; someone will still need to learn those tools and how they can be applied to expand a single experiment into a high-throughput one (still only covering one experimental design's portion of the underlying theoretical space). This question of implementation complexity is one practical concern among several that exist which lead me to conclude that, while the idea of high-throughput experimentation is aligned with an ideal scientific method, in practice we must be prepared to compromise in accordance with the practical considerations that delimit the boundaries of what a productive research group can accomplish in a reasonable amount of time. This is still limiting the consideration to more typical "social science" experiments; the story with e.g. fMRI research, which is my field of interest, is so much worse through this lens that, if the principles underlying the advancement of high-throughput experimentation today are virtual lab environments, machine learning, and mass collaboration, I believe their impressive progress development is merely a small fraction of that which will be necessary to allow high-throughput approaches to fMRI research in the future.

[isaduan]

Hi Prof. Almaatouq! Thank you for sharing your research. For your paper "high throughput experimentation", I wonder:

1. what you think of this experimentation approach in relation to large-scale group/social phenomenon e.g. hierarchy, war, cooperation, which seems expensive to experiment or even to simulate? how do we do more useful social science in those areas?
2. you noted that in previous studies, experimental conditions documented in a systematic way, but how can we leverage those data to contruct the experimental design space for future studies? do we ultimately want some ML predictive models of 'most promising research projects'?

[linhui1020]

Hi, Professor Almaatoq, welcome to our computational social science workshop. My question is that when you are conducting research about human behavior by using large data sets, have you ever done a complement study to actually involve human beings? Do the results show identical conclusions?

[GabeNicholson]

Hi Professor, thank you for coming to our talk. Replication failure is a serious issue in social science and your creative approach to solving the issue is much needed. I'm very excited to attend the presentation.

[Sirius2713]

Thanks for join our workshop, Prof. Almaatouq! Your ideas about high-throughput, adaptive experiment design are impressive. Very excited to attend your session!

[erweinstein]

Hi Professor Almaatouq! Many of us, including those who are first-years in the 2-year Computational Social Science MA program, had part of our lecture and readings this week on False Discovery and related issues. While I think it's very important that the idea from you and your co-authors of specifying/constructing and then sampling from the Design Space helps make explicit many issues about "Researcher Degrees of Freedom" that too often have been left implicit (at least until relatively recently when the full extent of the Replication Crisis became known), how would (or do) you make sure that high-throughput experimentation doesn't itself yield false discoveries or otherwise run into multiple comparison or Gelman-style forking paths problems?

[TwoCentimetre]

Sounds cool! Brilliant ideas of what I have never thought before. Looking forward to your pre. Thanks for sharing.

[ShiyangLai]

Hi Professor Almaatoq, I like your paper "high throughput experimentation" which is really ambitious. However, following my folks' concerns, I'm also thinking about the generalizability and implementability of the high throughput experiment method. Besides, high throughput experiment provides us with a full picture of the design space, though, how should we sample in this design space to make the study unbiased? Looking forward to your presentation.

[JadeBenson]

Thank you for sharing your interesting work with us! I think both these studies have the potential to change the way we conduct research and work together. I was curious about the qualitative aspect of the participants in the synergy study. On top of just improved efficiency, did groups experience more individual satisfaction with their results, enjoy the process, learn new things that they can apply to other projects, etc.? I think improved efficiency is obviously a very important metric of group synergy but I'd be curious about these other metrics that offer another way that groups might outperform individuals.

[william-wei-zhu]

Hi Professor Almaatouq, thank you for sharing your work. Looking forward to your presentation tomorrow.

[JoeHelbing]

You mention that running higher cost 'high-throughput' experiments leads to cheaper research in the end, since the data generated is of higher quality as a function of it's total cost, but assuming relatively fixed budgets for social science research, isn't this advocating even higher concentration of the issues around job access and the 'status effect' already present in social science? I'm not personally disagreeing on the value of 'high-throughput' studies, I just wonder as an aspiring graduate student, in a room full of aspiring graduate students, if what is gained is worth the cost.

[nswxin]

Thank you for coming！ Looking forward to seeing you soon！

[jiehanL]

Hi Professor, thank you for sharing this exciting study! I am wondering, how should high throughput experimentation - as you said, in which researchers run thousands of experimental conditions - be positioned in meta-analysis research, as it runs multiple experiments per study, should it be considered as possessing disproportional effect size than others?

[lguo7]

Hello Prof. Almaatouq, thank you for coming! I’m glad to attend your talk this week in person! I’d like to ask the motivations of creating virtual lab platform. Would you think that virtual lab platform would be more popular in the future?

[Qiuyu-Li]

Thanks so much for sharing your works Prof Almaatouq! Looking forward to your presentation!

[tangn121]

Hi Professor Abdullah Almaatouq, thank you for attending our workshop. You mentioned the role of group leader, which is really impressive. Looking forward to hearing more from you!

[DehongUChi]

Hi Prof. Almaatouq, thank you for sharing your study with us! I am very excited to hear more about your innovative approach!

[zhiyun0707]

Hi Professor Almaatouq, thanks for coming to the workshop! For the High-Throughout Experiment, I am wondering that other than benefits, are there any cons of implementing this experiment? Thank you!

[a-bosko]

Hi Professor Almaatouq,

Thank you for coming to our workshop! I found the paper "Task complexity moderates group synergy" quite interesting, especially in the understanding of task complexity. This study really emphasizes the need for strong, interacting groups in complex projects.

I look forward to your talk!

[LuZhang0128]

Hi Prof. Almaatouq, the idea of a high-throughput experimentation approach in behavioral and social science is really visionary. Looking forward to your presentation.

[Lynx-jr]

Thanks so much for sharing your works Prof Almaatouq! Looking forward to your presentation!

[ZHE-ZHANG-0213]

Thanks so much for sharing your works Prof Almaatouq! I am very excited to hear more about your innovative approach!

[Yutong0828]

Hi Professor Almaatouq, I really love your idea of high-throughout experiment! It used to be a concept widely used in drug discovery, but you and your collaborators were able to generalize it to social sciences, which is very innovative. Actually, I have also had similar confusions toward psychology research and theories. Even for a very mature research area, there would be tons of findings and theories, and it is difficult to tell which theory is better. Sometimes, because it's difficult to judge from the theory itself, other less relevant factors would come into play (like the status and impact of a specific researcher), and it might be an obstacle for younger researchers to attract attention and recognization. More importantly, the findings about a same topic can be fragmented and difficult to organize without frequent literature review/meta-analysis. However, because different researchers have different way of synthesizing studies, the review and meta-analysis can have large variance, which makes it difficult to form a holistic view of the area. Therefore, I really love the idea of high-throughout experiment and design space, through which we may be more aware of which stage a research area has moved onto, and then generate clearer opinion of where we are heading toward.

However, I believe one of the challenged faced by high-throughout experiment is the difficulty to conduct such large scale of experiment which can cover most of the experiment design space. So, I was wondering if machine learning and other techniques can be used here to help us reach our goal with fewer experiments. I was thinking that perhaps the algorithms can help us to infer and generalize based on fewer data points on the entire design space.

Another question is about the way to determine the experiment design space. Do you have any ideas about how we may develop a comprehensive design space? I believe a thorough summary on previous research can be helpful, but I would like to know if there is any other smart ways to achieve this goal.

Thank you very much! I am looking forward to your presentation.

[sabinahartnett]

Thank you for presenting at our workshop Prof. Almaatouq! I enjoyed your papers and found myself reflecting on my own experiences with group efficacy and innovation. I'm excited to hear your answers to many of my peers' questions. Especially about group heterogeneity and the role that various backgrounds (whether social/intellectual/economic/cultural/religious/academic) can play in innovation, communication and group cohesion.

[Dxu1]

Hi Professor Almaatouq, thank you for shaing your interesting paper! I am very intrigued by your paper on high-throughput experiment. I am interested in your take on the policy implication of adopting high-throughput experiment. In a one-at-a-time approach, people were able to get intuition easily on how a couple factors could affect the system. This simplicity seems to be quite important in guiding policy makers. In a high-throughput environment, where the complexity of the space increases drastically (and frankly, more difficult to derive an intuitive theory/conclusion given the difficulty in interpreting high-dimensional results), how do you see that researchers could bring the results to the policy realm and make impacts there?

[ValAlvernUChic]

Hi Professor Almaatouq! Thank you for taking the time to come to the workshop! Your paper on high-throughput experiments and their potential had me thinking about their possibilities. My question is quite simple - in your opinion, which social science discipline or general application would benefit most from this method?

[MengChenC]

Hi Prof. Almaatouq, thank you for presenting this innovative approach in the workshop, looking forward to your presentation.

[ChongyuFang]

Hello Prof Almaatouq, thanks for presenting your work in our workshop. I look forward to having an understanding about the high-throughput experiments you are about to present!

[YijingZhang-98]

Dear Prof. Almaatouq, thanks for bringing us excellent work. But to be honest, the materials are kind of beyond my knowledge. I'd appreciate if you could share some practical examples that applying the High-Throughput Experimentation.

[hshi420]

Hi Professor Almaatouq, I was wondering in which fields the research results can be applied. Looking forward to your presentation!

[chrismaurice0]

Hello Professor Almaatouq, excited to hear you speak with us tomorrow! I would specifically enjoy hearing more about your experimental design and how you controlled for different factors that could impact how individuals interact in a group with others.

[ginxzheng]

Hello Professor Almaatouq, thank you so much for coming to our workshop. Looking forward to your presentation tomorrow!

[yhchou0904]

Dear Professor Almaatouq, thank you for sharing your idea with us in the workshop. The goal of the high-throughput experiments is impressive, and this kind of method indeed meets the technical progress nowadays. I am wondering if you could share some research using this method, and what impact does it bring to the research?

[Raychanan]

Thanks for sharing your work with us!

[siruizhou]

Thanks for presenting this work. The high-throughput experiments seem to bring paradigm shift in social science researches. I am looking forward to learn more about it.