Meeting March 1

[JamesPHoughton]

Differences between survey and discussion questions

Discussion questions sometimes need to be different from survey questions. For example, we may want to parallel survey questions from ANES:

"Do you favor, oppose, or neither favor nor oppose banning the sale of semi-automatic “assault-style” rifles?"

Unfortunately, this isn't a great discussion question, as people tend to just say what they think, and move on without really engaging in the discussion as something to solve together. Instead, we can ask the question of the group as a whole:

"Should the US government ban the sale of semi-automatic “assault-style” rifles?"

What should we ask as a "leader's position" question?

[RachelAbigail]

Note:


We discussed and agreed on adding the two ANES questions verbatim to the pre and post survey. One of the two (re: use of force) is already covered by a discussion question. The other (re: urban unrest) is not, and we need to add a discussion question to get at the concepts it covers.

Initial suggestion for a discussion question that we can iterate on:
What’s the best way to deal with urban unrest? Is it better to use force to quell it or to address social injustice to prevent it?

[RachelAbigail]

GSS Survey Questions re: police

Are there any situations you can imagine in which you would approve of a policeman striking an adult male citizen? 
(https://gssdataexplorer.norc.org/variables/670/vshow)

Listed below are various areas of government spending. Please indicate whether you would like to see more or less government spending in each area. Remember that if you say "much more," it might require a tax increase to pay for it.
 C. The police and law enforcement.
 (https://gssdataexplorer.norc.org/variables/4059/vshow)

[JamesPHoughton]

Survey (pre/post)

• Use the literal form of the ANES (or GSS, PEW, other national surveys) questions in the Pre and Post surveys. (from https://github.com/willschulz/bad-influence/issues/1#issuecomment-1440314862)
• Don't use them in the discussion questions directly because they weren't designed to be discussion questions.
• Any additional questions that make sense given our discussion questions

Discussion prompts

• In the discussion want to make sure that we touch on all of the parts of the questions in the discussions so that we can interpret a null result as meaningful, and not because it wasn't discussed.
• New discussion questions:
  • Source of urban unrest?
  • Social injustice in policing?
  • Response to urban unrest?
  • rioting, using force, and maintaining law and order,

Tasks

• [x] @JamesPHoughton will fix bugs and push to the test server by Friday morning.
• [x] @xehu @RachelAbigail @dcknox @willschulz @christopherlucas will pitch in on discussion questions today
• [x] @JamesPHoughton and @willschulz will finalize discussion questions thursday afternoon and update on server.
• [x] Everyone can test in groups of 3 or individually, @JamesPHoughton will give instructions
• [x] @JamesPHoughton will send confidentiality agreement to CSS lab director + Duncan
• [x] @xehu will reach out to WBL for dry-run next week
• [ ] We are all stretching our writing muscles to get an MPSA draft together

[willschulz]

I'm having a hard time coming up with anything better than @RachelAbigail 's proposed "What’s the best way to deal with urban unrest? Is it better to use force to quell it or to address social injustice to prevent it?"

The ANES Q refers to "problems of racism and police violence," so my main suggestion would be to revise "address social injustice" to "address police racism and misconduct"

We still inherit a false dichotomy from the original ANES Q -- I guess my concern is that it's hard to see how someone would articulate a middle ground position on this question, but we can try it and see what kind of discussion it generates.

An alternative that sacrifices the "police racism" theme (which we already have a Q about) in favor of a truer dichotomy: "In cases of rioting and widespread public disorder, should police err on the side of using too little force (and risk letting people get away with crimes) or using too much force (and risk further inflaming tensions)?"

[dcknox]

Here's my attempt to synthesize our conversation. As noted above, we have the following three points to consider.

1. What questions will we ask in the pre- and post-survey?
2. What questions will we use to prompt discussion?
3. Do the discussion questions cover all aspects of the survey questions?

Below, I've pulled together what I think is a complete list of the proposals so far. In subsequent discussion, to keep things organized, let's try to refer to issues by index (e.g. A.1, B.2, C.3). Feel free to edit this post to add new points or modify the existing ones. If you want to add a note when editing (e.g. raise a concern inline in this list, or flag a change you've made), please add [FL: your note] to the end of that point (where FL is your first/last initials).

A. Survey questions

1. Overall, how positively or negatively do you feel about the police?
2. Do you believe police should be involved in greater or fewer aspects of public life?
3. To what extent do you believe racism is a problem in policing?
4. [VERBATIM ANES] How often do you think police officers use more force than is necessary?
5. [VERBATIM ANES] What is the best way to deal with the problem of urban unrest and rioting? Some say it is more important to use all available force to maintain law and order, no matter what results. Others say it is more important to correct the problems of racism and police violence that give rise to the disturbances. And, of course, other people have opinions in between. Where would you place yourself on this scale, or haven't you thought much about this? [EH: I feel like the last bit, 'or haven't you thought much about this?' sounds somewhat condescending ... but I suppose that's what's in the original?]
6. [VERBATIM GSS] Are there any situations you can imagine in which you would approve of a policeman striking an adult male citizen?

[RM: I like these questions. Assuming survey questions will be ordered randomly, but if not I would suggest 2., 3., 5., 4., 6., 1.]

---

B. Discussion questions

1. What’s the best way to deal with urban unrest (protesting, rioting) about policing? Is it better to use force to quell it? Address police racism and misconduct to prevent it? Something else entirely? [EH edits:

   • How should the police deal with urban unrest (protesting, rioting)?
   • How should the police deal with urban unrest (protesting, rioting)? Is it more important to quell it with force or to prevent it by addressing internal issues such as misconduct and racism?] [WS: Since my concern is that this Q is too open-ended, I EH's second suggestion, since it suggests 2 poles and drops the "something else entirely"] [RM: Agree on liking EH's second suggestion]

2. Do you believe officers should be able to use all available force to maintain law and order, or is it more important to reduce police violence? [EH: I really like this one!] [WS: Me too!] [RM: Agree]

3. Do you believe police should be involved in greater or fewer aspects of public life? [DK: I think this one works as a discussion question without modification, but please feel free to tweak] [EH: Alternative that focuses on feelings rather than police: "overall, how positively or negatively do you feel about police and their involvement in public life?"] [WS: I'd keep this "involvement" question as-is, and add "How positively/negatively do you feel about police?" as a separate, potentially introductory question] [RM: Agree with WS on keeping involvement question as is]

4. In what situations is it appropriate for a policeman to strike an adult male? Where should we draw the line? [DK: new proposal, please tweak] [EH: In what situations, if ever, is it acceptable for a police officer to use force against an adult male? ----- my main edit is to change 'policeman,' which is gendered, to 'police officer,' which is not. I also thought maybe we can say 'use force' rather than 'strike,' because I feel like 'force' is a broader term that might facilitate richer discussions and encompass different possible actions. I feel that I imagine strike as a strike using one's fist, as opposed to using a weapon, for example.] [WS: Agree with EH's edits, though I'm concerned this Q is too open-ended, but we can try it and see what people say] [RM: in this version, we have 3 of the 4 discussion questions touching on police use of force and we've dropped the racism question from discussion. I suggest cutting this one, keeping questions 1 & 2, adding "where should we draw the line" to 2, and adding back in the racism question, for our target total of 4 questions.]

[RM: in line with my ordering suggestion in A., I would suggest discussion questions proceeding as:

• Do you believe police should be involved in greater or fewer aspects of public life?,
• To what extent do you believe racism is a problem in policing?,
• How should the police deal with urban unrest (protesting, rioting)? Is it more important to quell it with force or to prevent it by addressing internal issues such as misconduct and racism?

• Do you believe officers should be able to use all available force to maintain law and order, or is it more important to reduce police violence? Where should we draw the line?]

  ---

  C. Are concepts invoked in survey questions discussed?

1. [X] Police overall [DK: implicitly covered?] [EH: I guess the 'greater or fewer aspects of public life' covers this; we could also ask the alternative I listed above about positive/negative feelings.]
2. [X] Role of police in society
3. [X] Appropriateness of force
4. [X] Urban unrest and rioting
5. [X] Addressing racism and police violence

[xehu]

Just left comments inline! Thank you everyone for your thoughts so far, and @dcknox for the great summary!

[willschulz]

Yes, thanks @dcknox for organizing this. I'm concerned that B1 and B4 are too open-ended, and we'll struggle to scale what people say along a left-right spectrum, which we need to be able to do in order to measure the polarization of the discussion. I imagine it would be particularly difficult to hand-code the left-right positions embodied by the "situations" offered in response to B.4, but perhaps this is the wrong way to think about it if we're going to use ML measures. So ultimately this is a question for @dcknox and Chris: should we be trying to craft discussion Q's that evoke utterances that a human following a rule-based coding scheme could easily place on a left-right line? Or can we confidently throw unstructured/idiosyncratic utterances at a language model and reliably extract meaningful political positions from them?

[willschulz]

@JamesPHoughton and I are going to start implementing the new Qs shortly -- per the conclusion of yesterday's meeting, we will err on the side of including more discussion questions than we ultimately want, so we can try them out and decide which to keep based on that experience.

[xehu]

Here is a summary of the feedback after testing with @ChristopherLucas and me:

1. [for both pre-and post-survey] The bullet points for the urban unrest question are weirdly off, and look like this:
2. [for both pre-and post-survey] The final boss question is missing, and there is a hanging 'loading prompt...' right before the submit button:

Here is a full-screen screenshot of the 2 issues mentioned above:

3. If someone fails to provide responses for the pre-survey question, they are still included in the discussion. This seems potentially problematic: what is our plan for people who do not submit pre- (or post-) survey questions? One potential solution is to drop participants who fail to answer any pre-survey questions within the time allotted, as they are nonresponsive. Since randomization occurs after the pre-survey is taken (presumably), we can safely drop participants prior to their being assigned to a condition. Otherwise, we should have a plan for dealing with missing or incomplete survey data.

4. It is possible to skip all survey questions, except for a select few demographic questions (the ones in the screenshot below, plus an additional question about birth year):

   

   We should consider making it mandatory to answer the questions related to our dependent variable.

5. In the post-survey, the last question seems best suited for an open-text: that is, the "are there any situations in which you would approve of a police officer striking an adult male citizen" seems to be more interesting if we ask participants to both answer yes or no, and provide an explanation of which situations, if yes, they imagine to be acceptable.

6. It seems like the time limit for responding to survey questions is too short. If we are giving people their payment codes until after, they should have plenty of incentive to finish them without the time pressure --- why not give people more space to think carefully about their pre- (but this applies especially to post-) survey questions? In the post-survey, nothing is waiting on them; so, why force them to respond to all the questions in only a minute or two?

7. In general, it seems that the discussion time is very short for these questions. Is this the final timing?

8. The question text seems too small, relative to the larger text and emojis below (“Group Free Discussion Time”) --- we should potentially add a header, box, or separator (either thin, regular, or thick!) to draw more attention to them.

[xehu]

Additionally, the false ... ternary? between urban unrest, racism, and police force still feels like a weird question to me when I read it. I know we already discussed this above, though!

[willschulz]

Summary of my test run with @dcknox :

• +1 The time is too short for each question (although the leader's time might be just barely enough, as long as they're prepared to go as soon as they get the bell -- so maybe making the questions bigger and giving some transition time between questions would help)
• +1 the question text is too small -- let's make the questions bigger than the "who has the floor" text, and maybe visually distinguish the elements by putting them in different-colored boxes?
• +1 "loading prompt" issue
• Our discussion ended abruptly before the appropriate time (probably my own error when updating the timings yesterday)
• The instruction "Group Free Discussion Time" is ambiguous, for various reasons we should drop the word "free" (ex. it implies they can talk about anything, not just the discussion q)
• We should reword the "somewhat easier/harder" response options to the "should it be made easier/harder for police to use force" Q, maybe to "slightly" or "a little bit" instead of "somewhat"
• What happens if the "time remaining" runs out in the pre-survey?
• Why do the pre-test questions show up after the waiting room stage? Wouldn't it be more efficient to have folks answer all pre-discussion survey Qs before being placed into the waiting room?
• In the post-survey, there are many questions that flow from the stem "others in the discussion" but the stem is small, at the top of the page and easy to miss. Probably better to repeat the stem in each question, and use formatting to aid readability
• We got sent back to the waiting room after completing the post-discussion survey

[RachelAbigail]

• For two of my tabs, had to refresh page to make political orientation questions show up at the beginning of the survey.
• +1 for pre-/post- discussion questions, would suggest "a little bit harder" "a little bit easier" for whether it should be easier/harder for police to use force, rather than "somewhat"
• +1 For pre-/post- discussion questions, there was a "loading prompt..." message above the submit button but nothing additional loaded
• +1 on needing to add a check so people can't enter discussion without completing pre-discussion questions
• +1 removing "free" and saying just "Group Discussion Time" instead
• For in-discussion questions, can we make the two parts of the urban unrest question the same size:
• +1 on urban unrest pre/post bullet issue:
• In the post-discussion questions, it said "thank you for completing the survey" and then asked me more questions. Should save the thank you for the very end.

[JamesPHoughton]

Issues taken care of:

• [x] increase question duration for both leader and other speakers
• [x] separate the questions from the "who has the floor" text and made them bigger and bolder
• [x] fix "prompt loading" error (arose because of a spelling mistake - we now have validators: https://github.com/Watts-Lab/deliberation-empirica/pull/453)
• [x] fix bullet point formatting (note to self - don't put blank lines between bullet points)
• [x] increase time on pre/post survey questions
• [x] fix discussion ended abruptly (arose because game stage duration was shorter than display time for some elements - validator for this issue is TBD: https://github.com/Watts-Lab/deliberation-empirica/issues/462)
• [x] "Group free discussion time" -> "whole group discussion time"
• [x] "easier/harder to use force" question should use "a little bit harder" instead of "somewhat harder"
• [x] de-specific'd the wording around "in the next section you will answer three questions..."
• [x] integrated stem "Others in this discussion..." into survey questions in postsurveys

Still to do:

• [ ] put in a "loading" indicator that shows before surveys are ready to display
• [ ] https://github.com/willschulz/bad-influence/issues/3
• [ ] https://github.com/willschulz/bad-influence/issues/4
• [ ] https://github.com/Watts-Lab/deliberation-empirica/issues/458
• [ ] https://github.com/Watts-Lab/deliberation-empirica/issues/349
• [ ] https://github.com/Watts-Lab/surveys/issues/127
• [ ] https://github.com/Watts-Lab/surveys/issues/119

[RachelAbigail]

@JamesPHoughton can you confirm the "still to do" tasks above have been completed and close the issue if so?

[JamesPHoughton]

Some of the technical issues identified here are still in play, but they are captured elsewhere, so I'm closing this to mitigate list proliferation.