search

Watts-Lab / surveys

Library of surveys for deliberation experiments
MIT License
3 stars 4 forks source link

[Add] Awareness of oppositional arguments #51

Open JamesPHoughton opened 2 years ago

JamesPHoughton commented 2 years ago

Perceived understanding of the topic

Measure the participants subjective assessment of their knowledge.

Examples:

Cassidy, Simon, and Peter Eachus. 2000. “Learning Style, Academic Belief Systems, Self-Report Student Proficiency and Academic Achievement in Higher Education.” Educational Psychology Review 20 (3): 307–22.

Beside each one of the techniques/topics listed below circle the number which you think most closely represents your level of understanding for that topic 1 “I have never heard of this.” 2 “I have heard of this, but don’t understand what it is all about.” 3 “I have heard of this and partly understand what it is about.” 4 “I have a reasonable knowledge and understanding of this.” 5 “I am very confident about my understanding of and use of this.”

Anson, Ian G. 2022. “Epistemic Confidence Conditions the Effectiveness of Corrective Cues against Political Misperceptions.” Research & Politics 9 (2): 20531680221107869.

Self-Assessment (SA) Battery Next we would like you to rate your own performance relative to everyone who has taken the test. Please use the sliding scale to evaluate your performance on a scale from 0 ("I'm at the very bottom") to 100 ("I'm at the very top"). A rating of 50 indicates that "I'm exactly average". [0:100]

How would you rate your performance on the test you just took? o Excellent (5) o Above Average (4) o Average (3) o Below Average (2) o Poor (1)

Anspach, Nicolas M., Jay T. Jennings, and Kevin Arceneaux. 2019. “A Little Bit of Knowledge: Facebook’s News Feed and Self-Perceptions of Knowledge.” Research & Politics 6 (1): 2053168018816189.

To determine each subject’s level of overconfidence, we subtracted the number of correct answers by the number of questions the subject believed they answered correctly, a standard operationalization in overconfidence studies (Moore and Healy, 2008: 502).

Out of 6 questions..

Rapeli, Lauri. 2022. “What Is the Best Proxy for Political Knowledge in Surveys?” PloS One 17 (8): e0272530.

A typical survey measure of self-assessed political sophistication is a simple question such as ‘how well informed would you say you are of political matters?’

Graham, Matthew H. 2020. “Self-Awareness of Political Knowledge.” Political Behavior 42 (1): 305–26.

Factual questions and certainty scale Certainty was elicited on a five-point Likert scale. Before both the general knowledge and partisan questions, respondents who stated their certainty level were first instructed to view the scale points in the following manner:

  1. Don’t know: you are making a pure guess.
  2. Not too certain: you are willing to guess, but you aren’t too sure.
  3. Somewhat certain: you are making a good guess, but it might be wrong.
  4. Very certain: you think you know, but there is a small chance your answer is wrong.
  5. Absolutely certain: your answer is definitely correct. During the survey, respondents were reminded of these instructions about how to use the scale. Each time the certainty scale appeared, the following labels were used:
  6. Don’t know (pure guess).
  7. Not too certain.
  8. Somewhat certain (good guess).
  9. Very certain.
  10. Absolutely certain (definitely correct). Table C13 lists the wording for each question. Most questions had explanatory text that appeared before the actual question text. Text in brackets indicates differences how the text differed between the [no partisan cues condition / and the partisan cues condition]. Answer choices are listed in the following order: correct, middle, least proximate. In the survey, all quantitative answer choices defined ”about the same” as either less than a 10 percent or 1 percentage point change.

Ortoleva, Pietro, and Erik Snowberg. 2015. “Overconfidence in Political Behavior.” The American Economic Review 105 (2): 504–35.

How confident are you of your answer to this question? • No confidence at all • Not very confident • Somewhat unconfident • Somewhat confident • Very confident • Certain

[Cole, Michael J., Xiangmin Zhang, Jinging Liu, Chang Liu, Nicholas J. Belkin, Ralf Bierig, and Jacek Gwizdka. 2010. “Are Self-Assessments Reliable Indicators of Topic Knowledge?” Proceedings of the American Society for Information Science and Technology 47 (1): 1–10.]

How familiar are you with the topic of this task? (1-Not at all, 4-Somewhat, 7-Extremely)

Personal importance / Issue importance

Ryan, Timothy J. 2014. “Reconsidering Moral Issues in Politics.” The Journal of Politics 76 (2): 380–97.

How important is this issue to you personally?

  • Not at all
  • Slightly
  • Moderately
  • Somewhat
  • Very

Personal Relevance

Ryan, Timothy J. 2014. “Reconsidering Moral Issues in Politics.” The Journal of Politics 76 (2): 380–97.

How important is this issue to you personally?

  • Not at all
  • Slightly
  • Moderately
  • Somewhat
  • Very

Interest in the topic:

Is this a moral issue?

Ryan, Timothy J. 2014. “Reconsidering Moral Issues in Politics.” The Journal of Politics 76 (2): 380–97.

To what extent is your opinion on this issue... ... a reflection of your core moral beliefs and convictions?

  • Not at all
  • Slightly
  • Moderately
  • Much
  • Very much

... deeply connected to your fundamental beliefs about right and wrong? [Same response options]

... based on a moral principle? [Student sample only] [Same response options]

Awareness of legitimate rationales for own and for opposing views

Survey Source

Survey Overview

Awareness of Rationales for Own and Opposing Views. “We are interested in hearing about the reasons people have for [supporting different presidential candidates/ favoring different sides of this issue] Regardless of your own views, what reasons can you think of for [supporting Bill Clinton/Bob Dole for President] [keeping responsibility with the federal government/transferring responsibility to the in- dividual state governments] [favoring/opposing affirmative action programs]?”

To tap the dependent variables for the first and second hypotheses, awareness of legitimate rationales for own and for opposing views, open-ended questions were used to solicit issue-specific rationales for three separate controversies including preferences among the 1996 presidential candidates, opinions about affirmative action for women and minorities, and opinions about state versus federal control of the welfare system.10 Randomizing the order in which own and opposing view questions appeared,11 respondents were asked what reasons they could think of for the various viewpoints. The open-ended responses were coded into individual rationales by two independent coders. To produce an indicator of awareness of rationales for opposing viewpoints, respondents were asked, “Regardless of your own views, what reasons can you think of for. ...”12 In other words, they were asked to view the issues through the eyes of the opposition.

As expected, the number of rationales that people could give for their own positions were, on average, significantly higher than those they could give for why someone might hold an opposing view (p < 0.001 in all three cases). As shown in Table 1, the measures of rationales for the two sides of a given issue were, also not surprisingly, significantly correlated with one an- other, thus indicating general knowledge of or interest in politics or in these specific issues. Three issues were used to get a broader sense of a given person’s knowl- edge of dissimilar viewpoints than one issue alone would make possible and were then combined into two additive indices representing a person’s overall aware- ness of rationales for oppositional views and overall awareness of rationales for their own viewpoints.

image

Analyses showed no significant difference between the two order- ings in the number of rationales offered for either own or opposing positions, and thus the order variables were dropped from further analyses.

Volunteered rationales for own and opposing views were taken at face value and not evaluated by any external standards of so- phistication. But coders did eliminate from their counts rationales that served to delegitimize the other viewpoint. For example, if a re- spondent explained why others supported Clinton with reference to negative personal traits of the opinion holder (“Other people might vote for him because they are stupid”) or negative traits of Clinton (“He’s so slippery and slick and a good puppet”), then these were not counted as acknowledgments of alegitimate basis for the oppositional viewpoint. To test the reliability of coding, two independent coders both coded a subsample of 105 of respondents’ answers to the open- ended awareness of rationales for own and opposing views questions. The correlation between the measures produced by the two coders ranged from 0.74 to 0.89 across the six open-ended questions, with an average of 0.80. For respondents who held no opinion on a given issue, their rationales for the two questions were divided equally between awareness of own and oppositional views.

Notes

Coding responses on this one is going to be time-consuming.

Tasks