✅ Turker survey

[JamesPHoughton]

I'm planning to run a survey with turk workers from our panel to learn about how workers use turk.

Primary questions for this survey:

• What prevents people from joining a videocall study?
• What might prevent people from completing a videocall study?
• Are there systematic biases that we introduce by recruiting only willing participants that may limit our generalizability?
• What can we do to encourage more people to participate?
• How should we coordinate recruitment?

Secondary questions

• are there other threats to validity we should have on our radar (distracted participants, etc)

What I don't have yet:

• way to ask respondents what they think would be the best way to recruit for simultaneous games

Can you check for:

• questions are clearly worded
• questions are "requested", ie, soft required
• open response validations make sense

Survey link:

https://upenn.co1.qualtrics.com/jfe/form/SV_5uS1z1V4RsQ6mh0

[markwhiting]

I heard some people created some lists of turkers that had VR. Admittedly, I totally forget who did this, but I'm sure I can track it down. I wonder if a similar dataset exists for people who are willing to join on video.

I also wonder if you had a HIT that showed people your requirements and asked them to specify why they might not want to join, would you learn something useful from their responses?

[markwhiting]

Looks like there are two "other" inputs for "What percentage of the HITs you work on do you find through each of the following sources? (please give your best estimate, total must sum to 100)"

(perhaps that's intentional, but if it is, its not totally clear that it is intentional, e.g., Other source 1, Other source 2)

[markwhiting]

Looks like you're using markdown for bolding but it's not showing up in qualtrics. If you'd like to, you should be able to do something like "rich editing" to make it do bolding.

[markwhiting]

Would any of the following be obstacles to participating in an audio-only conversation?

and the Video version are Checkbox then Radio response. Not sure if thats intentional.

[JamesPHoughton]

• Maybe we should ask whether people are "professional turker". Ask "do you have another job" professional turker question
• Can people describe the filters they use to decide what tasks to work on?
• Is there a social desirability bias that people will answer questions the way they think they should behave?
• We could run a 1-question survey over a larger part of the panel just about willingness to engage in a discussion about a politically contentious topic.
• Our discussion is not just about a video chat, it's about a potentially politically contentious topic. Add a question about whether they would want to have a discussion about a political issue.
• May be a good idea to post about this HIT on forums, etc.
• We may want to select a subset of people to make eligible, and then work really hard to recruit them, so we don't get a subset that is biased towards people who do a lot of HITs.

[JamesPHoughton]

The purpose of the survey was to determine whether the people who are willing to participate in a video discussion are systematically different from the people who are not.

Setup tasks

• [x] (James) Share qualtrics survey
• [x] (James) Share panel datafile
• [x] (James) Share google drive folder

Analysis tasks

• [x] Get the data from qualtrics, clean out test submissions
• [x] plot raw results - help us understand what is going on
• [x] plot correlations within the survey data specifically for willingness to engage in a video discussion with other participants
• [x] merge rows with panel datafile, drop participants that are not in both tables
• [x] plot correlation against for willingness to engage in a video discussion with other participants:
  • RME
  • CRT
  • gender
  • birth_year
  • education_level
  • country
  • political_party
  • race
  • trust_in_others

Constraints

• [ ] We will collect more data this week (currently have ~30? datapoints), so set up the analysis script so that its easy to rerun with a new data download
• [x] Set up as a jupyter notebook on colab, in Google Drive / CSSlab / Research / Integrative Experiments / Deliberation / Turker Survey
• [x] Ideally done by Wednesday evening at the latest

[JamesPHoughton]

A few notes reading through the notebook:

• [x] Before we drop rows with missing data, first create a table with only rows from MTurk workers it'll be a 10+ish character string of capital alphanumeric characters with no spaces that starts with 'A'.) and just the column responses we're using. This is the data we care about, so this can be our "working" dataset. Now drop rows with missing data, and print the number of rows in the working dataset before and after dropping incomplete rows. This means we don't drop anything because of data that is missing that we don't care about anyways, and also that we are computing the fraction of incomplete responses compared to an accurate baseline.
• [x] plot a histogram of the time duration for the working dataset, to get a visual sense for how many people rushed through the survey or dawdled.
• [x] For the devices, i don't particularly care about the combinations of who has everything so much as the percentage that has a webcam, the percentage that has a mic, etc. Can you simplify those bar charts? Same with the other multi-select questions.
• [x] Can you plot the 'task willingness' plots on a consistent set of axes? Probably what would be good would be a stacked bar chart (the bars should sum to 100%) with one bar for each task, and colors in the stack representing the fraction in each category. An you also label the tasks with the task name instead of number?
• [ ]

[JamesPHoughton]

Would also be good to do some more exploration specifically of the group that said they would be likely or very likely to participate in at least on of the video discussion questions with the group as a whole. Just do the breakdown into those two groups and plot all of the exploratory stuff again with the subset. You can probably combine the plots to show them both together.

Compare the two groups (willing and unwilling) on these dimensions

• RME
• CRT
• gender
• birth_year
• education_level
• political_party
• race
• trust_in_others

[JamesPHoughton]

Created a qual for participants in our panel who meet the following:

    panel["RME"].notna() & 
    panel["CRT"].notna() & 
    panel["gender"].notna() & 
    panel["birth_year"].notna() & 
    panel["education_level"].notna() &
    (panel["country"] == "United States") &
    panel["political_party"].notna() &
    panel["race"].notna() &
    panel["trust_in_others"].notna() &
    panel['BFI_conscientiousness'].notna() &
    panel['BFI_open_mindedness'].notna() &
    panel['BFI_agreeableness'].notna() &
    panel['BFI_extraversion'].notna() &
    panel['BFI_negative_emotionality'].notna()

This is only about 146 people, so I invited them all to our survey. Given our previous rate of about 20% who say they are willing to do video conversations, this should give about 30 people in the "willing" category, if everyone participates (they won't), which is right on the edge of how many we'd probably need to make reasonable comparisons between the willing and unwilling. We'll probably have to recruit some more participants for whom we have less panel data, once we see how this turns out.

[JamesPHoughton]

Some more comparisons:

Collapse the responses into two groups:

1. Those who are "Probably Yes" or "Definitely Yes" or "Unsure" if they are willing to participate in any form of video call with another worker. Use an OR operator to compress "a video call with 1 other MTurk worker" and "a video call with 2 or more other MTurk workers" into one column.
2. Everyone else, (ie. those expressing unwillingness)

Our goal is to figure out what is different between these groups.

Is it "technical ability"?

One thing that could be going on is that people aren't set up to do it. To be able to do the experiment, you need

• headphones/earbuds
• a webcam
• a microphone
• high-speed internet
• a desktop or laptop computer
• more than 45 minutes of uninterrupted time
• a room to yourself
• time specifically set aside for mturk OR between paid tasks when you can predict when the other work will take your attention OR during scheduled downtime

Lets aggregate all of these factors into a single column for "technically Able", and plot the fraction of people in each group (willing, unwilling) that are able. If a significant fraction of unwillingness can be ascribed to not having the right tools, this could mean that we might be able to work with participants to help them set up the conditions they will need to participate.

Is it one or more of their personality characteristics

Plot a histogram of the following values for each group, and give the mean differences:

• RME
• CRT
• gender
• birth_year
• education_level
• political_party
• race
• trust_in_others
• each of the BFI measures

What reasons do they give?

Within the group that expresses unwillingness, what fraction give each of the provided reasons for being unwilling?

[JamesPHoughton]

@mykoal - yesterday we were talking about comparing the extroversion, social trust, etc. of the "willing" group vs the "unwilling" group. Can you add an additional group, which is just the population of the entire panel for whom we have data? Not limited to the folks who returned the survey, but for everyone, so that we can see which group is more likely to be representative of the population as a whole? I'm sure it'll be the "unwilling" group because they make up a larger fraction of the whole population, but would be good to plot.