01/30: Becker

[jmausolf]

Comment below with questions or thoughts about the reading for this week's workshop.

Please make your comments by Wednesday 11:59 PM, and upvote at least five of your peers' comments on Thursday prior to the workshop. You need to use 'thumbs-up' for your reactions to count towards 'top comments,' but you can use other emojis on top of the thumbs up.

[wanitchayap]

Thank you so much for sharing this very interesting paper! I would really like to see the direction of your replicated experiment's outcomes. I think your suggestion to employ both the Delphi method and unstructured discussion for long-run/optimal benefit and lowering the risk in unknown situations makes a lot of sense to me.

My question is: Your model and suggestion seem to primarily consider only one cycle of individual-then-group decision process. However, in real life, especially when the issue to make a decision on is very significant (and with some time to decide), we tend to have unstructured discussion more than once. That is, we may start with individual decision, then having discussion, then ungrouping to rethink individually, and then having discussion again, so on and so forth. How would your model handle the group decision made by this scenario? Would your hypotheses still hold in this scenario? How would you account for possible changes in the power dynamic between different rounds of discussions?

[adarshmathew]

Thank you for presenting your paper at our workshop. The Partisan Priming experiment in your 2019 PNAS paper seems fascinating.

1. In your choice of the DeGroot model of belief updating, you don't consider the possibility of balanced cliques forming (multiple components), which would prevent consensus-formation dragging the group towards the truth. Could you explain why this generative mechanism (cliques) was ruled out in your theoretical setup?

2. I understand your argument about how talkativeness could serve as a proxy for influence within the discussion. But that doesn't account for the possibility that certain participants may choose to ignore the talkative node. How are you controlling for preferential attachment?

[anuraag94]

Thanks in advance for your presentation, and for sharing this interesting draft with us.

@adarshmathew brings up a valid critique of talkativeness as a proxy for centrality -> influence in unstructured discussion. I'll turn my attention to a related question that you asked in the paper: "who are central individuals?"

In the cases that you tested, you found that individuals who are more talkative are not more accurate. This is not suprising, since expert knowledge doesn't exist for estimation tasks like counting gumballs. Everyone involved in the task is an epistemic peer – in other words, everyone has exactly the same ability to answer the question at hand.

However, in order to decisively prove that emergent centralization determines group belief formation, it's not enough to say that everyone's on a level playing field. What matters is whether the subjects themselves believe that they are all epistemic peers.

If they instead believe that talkativeness is a signal of privileged knowledge, then they will assign more than equal weight to the views of talkative individuals when encountered. For example, in the case of estimating the length of the border of Switzerland, talk may take a pseudo-evidential turn, and some subjects might believe that others are more justified in their estimations.

How do you account for this?

[zeyuxu1997]

Thanks for presentation, your conclusion is heuristic and interesting. I wonder if you could apply your model to explain the results of opinion survey about Presidential Election and the final results of the election. For example, both opinion survey about the election in 2016 and media propaganda support that Hillary are more likely to win. However, we have Trump as the president now. Do you think that's because the discussion is not centralized enough, or there are some other factors contributing to the contradiction?

[WMhYang]

Thank you very much for the presentation. The conclusion seems surprising to me at first but becomes reasonable after thoughts. My question is also about the measurement of centrality. Since Delphi method is anonymous while unstructured discussion is not, talkativeness may be more connected with centrality in Delphi method because people have fewer ways to become the center. Therefore, there might be inconsistency when you are trying to employ the same proxy to different methods, especially when you are testing Hypothesis 2. Will this inconsistency matter in your analysis? Furthermore, although the result of table 2 is not significant, are the significance levels of Delphi method and unstructured discussion different?

[nwrim]

Thank you for the presentation and very interesting research! I think these results alone are highly interesting and look forward to seeing the replication dataset and results after the paper is published.

One question that arises to me when reading your article was that all the theories and discussions presented in the paper sound like there is an underlying assumption that all people have the same or similar level of domain expertise on the problems to solve. In other words, all people have roughly equal possibilities to get the right answer. In this case, it makes sense that "the proportion of individuals on the truth side of the mean" (or phi, as later discussed in the article) is indeed "the probability that the central node will fall on the truth side of the mean" (p. 9)

However, I feel like that in many situations, there are differences in domain knowledge among people in the group decision-making process and this domain knowledge sometimes makes a person central in the group. For example, if we have to guess the total number of faculties in the Philosophy department of UChicago and we have a person who majored philosophy in UChicago during undergraduate, I think he or she or they will definitely be more central to our discussion. And because of the a priori knowledge he or she or they have, this will increase the likelihood of getting the correct answer. In this hypothetical setting (which I think could be quite ecologically valid as well - one example is a lab setting where the PI has much more research experience than the rest of the lab), maybe the probability that the central node will fall on the truth side of the mean is not very related to the phi. Also, centralization will actually increase the probability of getting the correct answer in this case.

Do you think your theory could be generalized to the setting where there is a difference in domain knowledge? I confess my question resides in the shadow of the 1957 film 12 Angry Men - Juror 8 indeed is one of my heroes and I sometimes yearn for an opinion leader because of that.

[bhargavvader]

Hey Joshua, this was great - just as I started reading the paper, I thought, "oh look, another collective intelligence paper, didn't they have a bunch of contradictory results?", and then your paper goes on to discuss exactly this issue. It makes sense, intuitively as well, that the initial structure is very important in determining which method of collaboration would result in what kind of results. @nwrim makes a good point about how certain individuals having prior knowledge might also skew this - I'd imagine that knowing each other's confidences in the network might also change how the interactions and results happen...

This summer at SFI, I was lucky enough to meet/work with one of your friends and co-authors, Douglas Guilbeault (and our very own @bakerwho!). We researched how we can maybe measure collective associations of words to color by aggregating google search image results. For example, different emotions had different color associations (sad -> blue, angry -> red) based on the google images, and these clustered in a significant manner. I'm not sure if Doug told you much about this research, but I believe it combines both ideas of multi modal cognition (we think using multiple semantic paradigms), as well as a group cognition, where sharing of information shapes the way we think.

What do you think of how shared views of abstract concepts might emerge? How much do you think that existing network structure might effect it e.g many folks using Shakespeare's descriptions of emotions as colors?

[bakerwho]

Hello and thank you for presenting. It was a pleasant surprise to find out that you have co-authored with Douglas Guilbeault, a good friend from the SFI summer school.

My question is about how a committee-like environment may have one of two outcomes: a modal 'best estimate' (that presumably has nice statistical properties), or a multi-modal split, analogous to the birth of partisanship. Especially in your modeling of questions with numerical answers (such as height/weight ratios), I would assume an early set of estimates could crystallize easily into two divergent ones.

One would hope that an understanding of this one-dimensional case could inform the analysis of complex partisan splits like those observed in politics today.

Do you have any ideas on how to quantitatively identify this bifurcation? Multimodality is a tricky topic that I know Doug has worked with. I'd love to hear your ideas on this!

[policyglot]

Hi Joshua, thanks for this cutting-edge exploration, including extensions of work by some former speakers at this very workshop like Abdullah Almaatouq. I love that you do not assume familiarity with jargon and spell out meanings of each term clearly and concisely throughout your paper.

How would your model explain the findings of class social psychology papers like Asch (1951) where the participants caved into peer pressure from complete strangers and changed their opinion on an obvious visual display? Most vector centrality measures wouldn't apply here. https://www.simplypsychology.org/asch-conformity.html

Asch, S. E. (1955). Opinions and social pressure. Scientific American, 193(5), 31-35.

[liu431]

One motivating example you use in the numeric estimation tasks is that a marketing team estimates the sales demand for a potential new product. However, companies now are building algorithms and software at scale to make these decisions automatically, rather than people make decisions and repeat the process. As a result, how do you think about the impact of algorithms on collective intelligence and the accuracy of group beliefs?

[vinsonyz]

Thank you for your presentation! Actually I am so interetsed in that how would the network play a role in the group decision making process in terms of peer pressure? Since it might be helpful to study the performance of students and employees.

[lulululugagaga]

Thank you for your presentation. This is an interesting study of group beliefs. My question is how might you balance the mutual influence/interaction of in-group and out-group effects when forming and participating in a group?

[YanjieZhou]

Thanks very much for your presentation! It is really a appealing topic on how mediated processes can exert different influence on people's beliefs compared with unstructured communication using network approaches. Personally I have read some literature on network and am especially interested in hierarchy. Do you think that hierarchy plays a significant role in the effect of both unstructured communication and mediated processes?

[linghui-wu]

Thank you, Dr. Becker, it is indeed a great job. There is no doubt that I am really glad to see the consistent empirical patterns resolve the contradictions in the previous studies. My question is somewhat like that of @wanitchayap. Sometimes we may need to discuss the same project with different groups back and forth, so the centrality may change accordingly. Will the conclusions hold true if we incorporate the dynamic nature of the group discussions?

[timqzhang]

Thank you for your presentation. I am a bit doubted on the hypothesis that talkative people are more likely to have larger impact in group. I wonder if you have proper control on it since obviously the leadership for a team or group is decided by multiple factors. The similar question may also pose on the select of proxy. Besides, what is the application of this research, since it may not conclude an explicit implication, thank you !

[goldengua]

Thanks for your presentation. I was wondering how the discussion dynamics changes as a result of after-discussion interaction and networks, and how this might influence the follow-up discussion, as wanitchayap points out that there is unlikely to be only one discussion.

[rkcatipon]

Dr. Becker, thank you very much for sharing your research. Like others have discussed, I am also interested in better understanding the underlying assumptions that guide your hypothesis. To confirm, is there an assumption that all discussion participants have the same level of knowledge and an assumption that there all members are treated equally within the group? What motivated these assumptions?

Separately, as the article posits talkativeness as akin to influence, how can firms optimize decision making to include valid input from less talkative people?

[SoyBison]

Thanks for presenting your research. I was wondering if you elucidate the theoretical mechanism from a causal perspective. Based on your findings, how does the process you studied cause the results that you find, and what additional assumptions are needed to construct that causal structure?

[chiayunc]

Thank you for sharing your work today. My question is regarding the jury system in common law jurisdictions. In the paper, you mentioned that for unstructured discussions, the outcome is heavily influenced by the initial belief distribution. For jury discussions, although the facts and laws are discussed in legal proceedings in a very structured way, the jury enters deliberations to reach a verdict without any moderation. If in fact, the verdicts are reached in a way that is highly susceptible to the initial belief distribution, would you find this potentially problematic?

[ydeng117]

Thank you for your presentation. One question I would like to ask is that does authority play an influential role in forming the belief within a network? Max Weber characterized three types of authorities, charismatic, traditional, and bureaucratic. Would different kinds of authority within a certain network yield distinctive results in the belief?

[anqi-hu]

Thank you for sharing your work with us! I noticed that you did not address/ discuss the sizes of these decision-making groups. Intuitively speaking, I suspect that as the size of a group increases, the centrality of one individual might become less pronounced as other people's opinions are introduced into the conversation. As others have mentioned above, multiple modals of leading opinions as well as cliques of individuals could form within a group.

My question is: do you think it is the same case when centrality shifts from the individual level to the clique level? i.e., instead of a single individual's, a clique's opinion becomes the leading belief that shapes the larger group's decision?

[RuoyunTan]

Thank you for the presentation. In your paper, you mentioned that one limitation of the study is that the Delphi method used in the experiments are all about numeric beliefs, but numbers are not the only things that participants share in applications of the Delphi method. Could you elaborate more on how we could design and conduct experiments to study different scenarios? What may be different from the existing experiments that you analyzed?

[AllisonXiong]

Thanks in advance for presenting your inspiring research! Your effort on resolving contradictions among previous research by revealing two interating factors is impressive and persuasive. I have two questions on the measurement of network dynamics:

1. Numeric information and estimation were used in the experiment. However, the main outcome is merely a binary metric that recorded whether the belief is more accurate. Do you think the precision of the study would be further improved if the extent/amount of belief change is taken into account?
2. As already mentioned in the paper, talktiveness alone doesn't seem like the best factor of individual influence. Chatty people or people who gabble a lot on a single topic are not necessarily influencial. The number of people with whom one interact could also be taken into account.

[ZhouXing-19]

Thank you for the presentation. I have a small question concerning the group discussion. When it comes to the estimation of the effect of discussion on decisions, teammates who cooperates for a long time may have convergent ideas than those in a fresh new group. I wonder if the age of the group may need to be taken into consideration? Thank you!

[Yilun0221]

Thanks for the presentation! My questions mainly come from the following two aspects: 1. If the members of the selected team are not talkative, how to conduct further research? 2. How to measure the impact of the formed collective belief on the team positively or negatively?

[di-Tong]

Thanks for sharing this interesting project! Your paper powerfully identifies several key factors for decision-making and resolves contradictions in previous research. I wonder to what extent do decisions in real world practices fit into the general statistical and network model analyzed in this paper? How could you assess it under different contexts?

[hesongrun]

Thank you so much for the paper. What's the real world implication for your research? As I can see from your paper the formation of opinions is a complex process within a group. However, in real world, when should we encourage discussion and when should we encourage personal meditation? Looking forward to your presentation.

[lyl010]

Thank you for your presentation! Your analysis reminds me of a board game where a small group of people will be assigned to a type of character to tell lies and others need to decide who are telling lies. Everyone should say something in a turn, and it turns out that the 'unstructured' communication can be super confusing with contradictory facts at the beginning. And I am interested in the unstructured discussion and am wondering what leads to the turning point that common opinion is heading towards or far away from the truth from the chaos? And what restrains the truth from coming out in a unstructured discussion even when the proportion towards truth is beyond 50%? Thank you!

[JuneZzj]

Thank you for presenting. It is interesting to see how the centralization and people's idea changing with the group discussion. I noticed that you applied hierarchy logistic model for estimation. What is the reason for using the logistic regression when studying the dynamic social network. What is the strength and weakness of this method. Thank you.

[sunying2018]

Thanks for your presentation! I have a similar question as @JuneZzj . You mentioned on page 15 that your main statistical test is a logistic regression to predict the likelihood that a group will improve. I am wondering the reason that you choose this method, instead of other regressors in ML? Thanks!

[MegicLF]

Thank you for raising such an interesting topic! My understanding is that both the topic of the discussion and the members of the discussion would have an impact on the relative benefits of a discussion. My question is that how you would estimate the influence of the experiment design in the previous works on both your and other researchers’ analysis.

[tonofshell]

Thank you for sharing your work! Increasingly more of our social interaction and collaboration is done online. Do you think having a conservation over digital messages would create different outcomes compared to a face-to-face discussion? How might your research help collaboration tools such as Slack or GitHub be more effective in driving collaboration through better conversations?

[dhruvalb]

Thank you for sharing your work - looking forward to the presentation! In the formation of belief, do you think there will be difference in the process in different global cultures? Why or why not?

[YuxinNg]

Thanks for sharing your work. I am also very curious about the social interactions/ discussions online as @tonofshell mentioned. I am wondering in the situation where people are doing online "face-to-face" discussion, like video chat, does what you found in your research still applies? Or more generally, does what you found in the research applies to all online discussions? Thanks!

[SixueLiu96]

Thanks for presenting your research! It seems to me that it's pretty interesting studying the communication between group members whether will improve the accuracy of estimations not. Because the accuracy of estimations can be measured in different ways. My question is do we have a universal measurement on how accurate the belief is and the way to improve it? Thanks!

[dongchengecon]

Thank you so much for the presentation! In the paper, it is argued that network theory could resolve the contradictory findings in previous literature. I am wondering if the analysis framework or the results could be applied to the jury decision making process. When the whole group is facing incomplete information or biased expression, and the aim is to get the right decision, what kind of mechanism should be adopted?

[hihowme]

Thanks a lot for your presentation! This is a truly interesting topic about idea changing between group members. I am wondering is there anyway we could see the conditions in real life, so that we could choose to be in a group or do it on ourself based on the model? Thanks a lot.

[PAHADRIANUS]

Thank you for sharing your progress on looking into the group structures and optimization in group functionality. I am really looking forward to your presentation. I am quite impressed that you managed to uncover so much only by retesting data gathered by previous studies on the subject. Given the limitations and lacking accessibility of these results, it seems extreme challenging to devise updated statistical tests and theoretical supports based on them. With these data you to quite some extent explored the mechanisms behind the group estimation accuracy and located some of the more important factors including the equal representation of all group members and the potential errors caused by over-representation of some central figures, I think there are massive further depth on this matter that can be dug into. Taking a look at the used datasets produced by previous researchers, I found them overall small in scale, narrow in feature extensiveness, and quite confined under laboratory conditions. Should you manage to conduct a more broadly constructed data collection process to document more behaviors that could potentially be mechanisms behind the group collaboration, there will definitely be even more substantial progress. In addition, I am a bit suspicious about the estimation questions designed by the previous studies as they include questions on geographic and demographic knowledge that are clearly not equally difficult for all test subjects. I am concerned that they may not generate a fair measure for groups accuracy.

[Leahjl]

Thank you for presenting your paper at our workshop. My question is about the dynamic of belief formation between subgroups. If there is a group consist of several subgroups, does the process also apply to this kind of situation? What is the network influence between subgroups?

[ShanglunLi]

Thank you for providing such an interesting paper to read! Can you explain a bit about how the network affects group decision-makings? Thank you.

[bjcliang-uchi]

In real life, we observe a repetitive information exchange process where, with feedbacks from non-centralized nodes, the total network structure evolves (some nodes become more trustworthy, etc). I am wondering how much this current model is robust in a time-variant situation.

[yutianlai]

Thanks for sharing your work! It's interesting to see how group discussion drives a centralized opinion. I’m wondering about the application of your research in reality. Many factors in real life might change your results, such as different cultures, different topics, or different group members. Is it possible to draw a more holistic picture? If so, how?

[KenChenCompEcon]

Thanks in advance for your talk tomorrow! This is a very intriguing topic that drills into the effect of group work on the ultimate quality of the collective prediction/assessment. I am curious about any insight into how future working teams should be organized to improve collective accuracy?

[Anqi-Zhou]

Thank you for the presentation in advance! Really interesting topic! My question is: How do you address other potential factors when you conclude the talkative people are less accurate?

[heathercchen]

Thanks in advance for your presentation! I have a question regarding the definitions in your article. As you mentioned in the hypothesis that group interaction and group intelligence may improve the process of decision making. But how do you define a solution or a group idea as "wise"?

[skanthan95]

Thank you for your presentation! Are there ways (a priori) to determine if the goal of a discussion is not consensus - for instance, if one or more answers are meant to sabotage the final answer. This seems like a vulnerability of the Delphi method, and has been observed with mindless responses to MTurk survey research.

[SiyuanPengMike]

Thanks for your interesting and inspiring paper. I'm a little bit confused that why you select logistic regression as your model's statistical tool. When calculating the effect of networking, MLP models usually outperform and are widely used. Could you please give us some hint of why you didn't use those more advanced neural network models?

[luyingjiang]

Thank you for your presentation on how network theories of belief formation can resolve these inconsistencies. I am wondering how network structures could adapt to the attributes of different individuals? Does this kind of adaptation promote the accuracy of individual and collective decisions? For example, are prestige members more influential and their beliefs weigh disproportionately?

[tianyueniu]

Thank you for the interesting draft! It was very inspiring to see a thorough analysis on the different results generated by Delphi & unstructured discussions. My question is, if the group is a high-context group in that the majority of the group members know who are the best judgement-makers & who talk a lot but aren't necessarily right, how would that change the result? Would group members' relationship change the model?

[caibengbu]

Thank you for the presentation in advance! It’s an interesting and novel topic! It really gives us insight on how future working teams should be organized to improve collective accuracy! Looking forward to it!