[Add] Willingess to return

[JamesPHoughton]

We want to measure whether participants would be willing to work with the same group again, asked in a behavioral way that makes it seem that if they say yes, we're going to assign them to actually do it. At the same time, we want to get a more nuanced view

Possibly a multi-part question, such as the US partisanship question Part 1:

Are you willing to participate in another discussion with this group of people?

• Yes
• No
• Not sure

Part 2

(For yes) How excited/interested would you be to join?

• I'd be excited to do it
• I'd be ok with it

(For no)

• I absolutely refuse
• I just prefer not to

(For maybe)

• Lean yes
• Lean no

Then compose these into a 6-point scale? Or just do yes/no to make it even more behavioral (ie, make it seem as if they are making a choice in the moment).

[markwhiting]

Cool. We had a behavioral version in the fracture work. We didn't get a nuanced rating of that, but we did run the viability instrument that correlated well with the binary response.

One thing we did in our case was something like "you're almost done, you could be selected for one more activity, do you want to work with the same team or a randomly selected one?". This could be augmented to be a scale. e.g., “if you get chosen to work one more round, would you prefer to:” "definitely work with the same team", "probably work with the same team", "probably work with a new team", "definitely work with a new team".

[JamesPHoughton]

Would you like to participate in this or similar conversation studies in the future?

• Yes
• No

If you answered yes to the first question, would you like to be notified of opportunities to participate in this or similar conversation studies in the future?

• Yes
• No

If you answered yes to the first question, would you like to have another conversation with the same group of people?

• Yes, I would prefer to rejoin this group rather than a different group, if possible
• Yes, I am equally happy to join a discussion with this group or a new group
• No, I would prefer to join a different group, if possible
• No, I do not want to join this group again

If you answered yes to the first question, would you like to have another conversation about the same or a similar discussion topic?

• Yes, I would prefer to discuss this topic again rather than a new topic, if possible
• Yes, I am equally happy to discuss this or other topics
• No, I would prefer to discuss a different topic, if possible
• No, I do not want to discuss this topic again

If you answered no to the first question, can you tell us why, so that we can improve?

• I do not want to have another discussion with this or a similar group
• I do not want to discuss this or a similar topic
• I do not feel I was adequately compensated
• Other

Can you tell us about what would make you willing to return to this or a similar study?

---

[JamesPHoughton]

Other uses in the literature: "Would you like to have aconversation with this person again in the future? (Yes/No)" (Huang, Karen, Michael Yeomans, Alison Wood Brooks, Julia Minson, and Francesca Gino. 2017. “It Doesn’t Hurt to Ask: Question-Asking Increases Liking.” Journal of Personality and Social Psychology 113 (3): 430–52.)

[markwhiting]

Francesca Gino!

[JamesPHoughton]

The rest of the authors are wonderful. =) This is just the world we play in now...

[JamesPHoughton]

notes

• Begin with highly "behavioral" questions that ask the participant about their desires in such a way that it makes it seem that they are making an imminent choice to participate in the conversation, rather than being surveyed on their opinions.
• Want people to answer based on their own desires, not on logistical constraints about their availability.
• Whether we include questions about preferences to continue with the same topic or discussion group in the same page as the question about participating in another conversation may change how people answer the first question, as they may be more likely to say "yes" if they can express a desire to have a different discussion group or topic.
• There is a potential for acquiescence bias if we frame answer choices as "yes/no", and so we may want to explore wordings that do not indicate that any particular choice is the correct one. On the other hand, we're interested in measuring willingness to return to the group and discussion topic, so that we can generalize to situations where a group must work together and maintain a working relationship over several sessions. This would imply that we should ask questions where returning to the group was the default, and not returning to the group was a refusal, rather than an equally valid option?
• Would be nice if we could aggregate to a more nuanced "willingness to engage" metric - possibly by summing answers to these questions (ie, if Q1 is "no", 0 points, if "yes" 1 point, plus 3 points for preferring to continue with the same group/topic, 2 for being ambivalent, 1 for having slight preference for other, 0 for avoiding the topic/group.)

questions

1. Would you like to participate in another conversation as part of this study, at a time that is convenient for you?

   • Yes
   • No

2. If you answered 'yes' to question 1, please indicate your preference for the discussion group:

   • I prefer to continue with the same group (if possible)
   • I am open to continuing with the same group or joining a different group
   • I prefer to join a different group (if possible)
   • I do not want to have any further discussions with this group
   • N/A (answered 'no' to Question 1)

3. If you answered 'yes' to question 1, please indicate your preference for discussion topic:

   • I prefer to continue with the same topic (if possible)
   • I am open to continuing with the same topic, or discussing a different topic
   • I prefer to discuss a different topic (if possible)
   • I do not want to have any further discussions about this topic
   • N/A (answered 'no' to Question 1)

<if Q1 == 'yes', display Q4 on a separate page>

4. Would you like to be notified of opportunities to participate in conversation studies in the future?
   • Yes
   • No

<if Q1 == 'no', display Q5 on a separate page>

5. Can you tell us why you are not interested in participating in further conversations, so that we can improve?
   • open response

[xehu]

I really like this idea! I'm not sure that I have a ton to add --- I think it's really cool to get a behavioral measure of how willing people are to have a new conversation with the same group, as opposed to a general measure of how this particular conversation went.

A few loose thoughts:

1. Getting people to plan for the conversation. Sort of inspired by planning prompts, we could expand the first question slightly by actually acting as if we may hold a second experiment on a specific date, and having people mark their calendar and indicate that it will be happening. Whether or not this happens, this would lend more realism to the decision --- making it seem legitimate.
2. Different levels of willingness for different group members. Sometimes, people may have different levels of willingness to work with different group members. Without making this too complicated, I think it might be interesting to give people the option to express a willingness to work with some people but not others (i.e., a choice for each group member / select which group members you'd like to work with again?)
3. With regard to scoring, I wonder if we should separately consider unwillingness as a result of a particular group (or group member), as opposed to unwillingness as a result of a particular topic. These things obviously interact, but it's unclear exactly how --- extremely pleasant people can make unpleasant topics easier to discuss (and vice versa).
4. Another idea I had could be reframing the behavior as sending an invitation, or a thank-you note, or other message to the other team members. I wonder whether that will be another way to create an "honest" signal of how much you enjoyed the conversation --- whether you liked it enough to actually take the time to write a message, or notify the other party that you'd like to talk to them again.

[JamesPHoughton]

Examples from the literature

Willingness to engage with disagreers/outparty in general

Santoro, Erik, and David E. Broockman. 2022. “The Promise and Pitfalls of Cross-Partisan Conversations for Reducing Affective Polarization: Evidence from Randomized Experiments.” Science Advances 8 (25).

"whether participants expressed a behavioral intention to engage with outparty. We measured this with three items: Participants were asked to indicate the extent to which they would be interested in having another conversation with a member of the outparty, if they would learn from such a conversation, and of the participants who indicated that they had a conversation, if they would be interested in meeting up with their partner again. We formed an index with all three items."

Kamin, Julia. 2022. “Social Cohesion Impact Measurement (SCIM) Framework Overview.” Civic Health Project. https://docs.google.com/document/d/1_nsLJNgWZVaNSq71PFpAHx7YM488FvTPIPFYWsytwus.

"How much do you agree or disagree with the following statements? (1) I am confident in my abilityto have fruitful conversations with the people that I hate/disagree with the most. (2) I can collaborate well with the people who I hate/disagree with the most."

Rossiter, Erin. 2023. “The Similar and Distinct Effects of Political and Non-Political Conversation on Affective Polarization.”

"In Study 2, participants were asked about their willingness to have future conversations with outpartisans. To assess willingness to have future non-political conversations, I asked about willingness to talk about "family," as it is akin to the non-political conversation prompt of meaning of life in the experiment. Likewise, to assess willingness to have future political conversations, I asked abouf "immigration," as that was the political topic prompt. Specifically, participants were asked to indicate their agreement on a five point scale from strongly disagree (1) to strongly agree (5) for the following questions: "I am willing to have a conversation about my family with a person that identifies as [Democrat/Republican]" and "I am willing to have a conversation about immigration with a person that identifies as [Democrat/Republican]."

Willingness to have another conversation with the same group:

Huang, Karen, Michael Yeomans, Alison Wood Brooks, Julia Minson, and Francesca Gino. 2017. “It Doesn’t Hurt to Ask: Question-Asking Increases Liking.” Journal of Personality and Social Psychology 113 (3): 430–52.

"Would you like to have a conversation with this person again in the future? (Yes/No)"

Rossiter, Erin. 2023. “The Similar and Distinct Effects of Political and Non-Political Conversation on Affective Polarization.”

In Study 1, participants were asked about their willingness to talk politics again with their assigned conversation partner, asking agreement on a five point scale from strongly disagree (1) to strongly agree (5) for the statement "I would chat with my conversation partner about politics."

Santoro, Erik, and David E. Broockman. 2022. “The Promise and Pitfalls of Cross-Partisan Conversations for Reducing Affective Polarization: Evidence from Randomized Experiments.” Science Advances 8 (25).

"whether participants expressed a behavioral intention to engage with outparty. We measured this with three items: Participants were asked to indicate the extent to which they would be interested in having another conversation with a member of the outparty, if they would learn from such a conversation, and of the participants who indicated that they had a conversation, if they would be interested in meeting up with their partner again. We formed an index with all three items."

Bayazit, Mahmut, and Elizabeth A. Mannix. 2003. “Should I Stay or Should I Go? Predicting Team Members’ Intent to Remain in the Team.” Small Group Research 34 (3): 290–321.

Intent to remain in the team. We measured intent to remain with 3 items focused on members’ desire to stay in or leave the team (i.e., “If I could have left this team and worked with another team, I would have,” “I wouldn’t hesitate to participate on another task with the same team members,” and “If given the choice, I would prefer to work with another team rather than this one.”). Items were rated on a 5-point, Likert-type scale ranging from 1 (strongly dis-agree)to5(strongly agree).

Bushe, Gervase R., and Graeme H. Coetzer. 2007. “Group Development and Team Effectiveness: Using Cognitive Representations to Measure Group Development and Predict Task Performance and Group Viability.” The Journal of Applied Behavioral Science; Arlington 43 (2): 184–94,196,198–202,204–12.

The specific questions used to measure satisfaction with membership and satisfaction with output were developed for this study. Satisfaction with membership was composed of three items: Being a member of this team has been personally satisfying, I would choose this team to work with on similar tasks in the future, and being a member of this team was a positive experience. Satisfaction with output was also composed of three items: I am satisfied with the final project of this team, we did an excellent job on our case analysis, and the team’s final paper is better than what I could have done on my own.

Willingness to have another conversation about the same topic:

Rossiter, Erin. 2023. “The Similar and Distinct Effects of Political and Non-Political Conversation on Affective Polarization.”

To assess willingness to have future political conversations, I asked about "immigration," as that was the political topic prompt. Specifically, participants were asked to indicate their agreement on a five point scale from strongly disagree (1) to strongly agree (5) for the following questions... "I am willing to have a conversation about immigration with a person that identifies as [Democrat/Republican]."

[JamesPHoughton]

Different stakeholders have different interests in making people more willing to return, so we want to ask both about re-engagement with the group and separately re-engagement with the topic, and also engagement with people they disagree with in general.

1. Bridging organizations are interested in getting people to engage more generally with people that they disagree with. This would mean that their work is helping to create dialog between cross-partisans outside the direct context of the intervention itself, either about the assigned topic or another topic of disagreement. In addition, the existence of the "contact theory" literature and its general agreement that just getting to know people who disagree with you helps them seem more human and less "other", we might be interested in increasing general willingness to have a conversation with an outpartisan as a general good in its own right.
2. Individuals who are in existing or nascent relationships care about ensuring a willingness to invest further in those relationships, even if the discussion topic never comes up again, because they care about building relationships with other people.
3. Managers or committee conveners care about ensuring commitment to ongoing productive discussions to achieve some outcome with respect to that topic, and so care about willingness to reengage both with the same group and the same topic.
4. Educators might be interested in encouraging people to reengage with a particular topic, to improve their learning about that topic, etc.

[JamesPHoughton]

One of the contexts we care about is willingness to engage in a conversation with cross-partisans or people they disagree with in the real world. This is subtly different from being willing to come back to a group or topic they have already started discussing in a particular context. We'll leave that question out of this part of the survey itself to be asked somewhere else.

To make everything in this survey super behavioral, we can ask very simple questions:

• Would you like to participate in conversation studies like this one in the future? (yes/no)
• Would you like to have another conversation with your discussion partner(s) in the future, about this or other topics? (yes/no)
• Would you like to discuss this topic again in the future, possibly with a different group? (yes/no)

These could be generalized for non-conversation studies that have groups:

• Would you like to participate in studies like this one in the future? (yes/no).
• Would you like to join a group with the same partner(s) in the future? (yes/no).
• Would you like to return to the same topic/task in the future, possibly with a different group (yes/no).

[JamesPHoughton]

Opening options:

• "Are you willing to..." : implies a request - i will ask you to do this
• "Are you interested in..." : maybe less behavioral. Implies an inquiry?
• "Do you want to..." : has some immediacy, most behavioral, implies a choice you are making now
• "Would you like to....": more inquiry? sounds like an offer they can take up or leave
• "Would you be willing to participate..."

temporal:

• (no indication): implies now? May get some negative responses because of logistics. If we put it into
• "in the future" or "in another session": maybe less of a commitment, as there is more temporal distance?

regardless of topic:

• no mention of topic
• "...on any topic?" - possible misinterpretation that they can choose the topic?
• "...regardless of topic"
• "...on a different topic" - may interact with question about topic, because if they say yes to talking about a different topic, they may not think they can also say in the next question that they want to talk about the same topic
• "... regardless of what topic is assigned?"
• "... regardless of the assigned topic?" - possibly interpreted as committing them to talk about any topic, giving up the ability to decline a topic
• "... possibly about a different topic?" - doesn't seem like it commits them to any course regarding the same or other topic

regardless of partner:

• "with any discussion partner"
• "possibly with a different group"
• "with a different group"

If we have multiple yes/no questions that are all choices "do you want to", do they compete with one another? Are they choosing between talking about the topic again OR talking with the same group again?

new questions:

• Do you want to participate in another conversation study like this one?
• Do you want to talk with the same partner(s) again, possibly about a different topic?
• Do you want to discuss the same topic again, possibly in a different group?

[JamesPHoughton]

Complete