build hit for turkers

[bellecarrell]

You can start to put together a Mechanical Turk HIT while you wait for users to be filtered. Attached is an example HIT form. We want to adapt this HIT to present the Turker with a link to a Twitter user summary page (example format: https://twitter.com/${user_screenname} ) and then ask them to annotate the following:

• Is this user still available, or are they no longer active? (checkbox)

• Based on their most recent tweet, does it appear that they use their account to promote their online presence, or it it mainly used for personal use/keeping up with friends and family? (checkbox)

• Mark which broad categories this user actually tweets about based on their most recent tweets (choose from categories we listed the other day along with Other). Also, select which category they are most focused on. If you are unsure, do your best. If selecting Other, please describe what Other category they focus on. (checkboxes to mark all categories, radio button for most-focused category, free text field for Other category description) -- You should include some examples of each category in the instructions for reference.

You can test it by deploying the HIT in the developer sandbox: https://requester.mturk.com/developer/sandbox

and perform it in the worker sandbox: https://workersandbox.mturk.com/

You want to make sure that all the fields we want to collect are included in the output CSV. When you think it looks good, point me to the deployed HIT along with the output CSV.

(check email for html file)

[bellecarrell]

started looking at developer sandbox

[bellecarrell]

sketched an outline of the HIT -- will make page Sun or Mon

[bellecarrell]

added example file and current version of hit to repo under uis/hits. still need to

• [ ] modify instructions

• [x] show/hide 2 and 3 based on whether user is active

• [x] make sure other textbox works

• [x] form submission

• [ ] anything else

  @abenton could you take a look and let me know if:

1. the descriptions sound okay
2. the font for the options is large enough (and any other aesthetic things to fix)
3. if there's anything missing, big or small

[abenton]

HIT Feedback:

1) The text in the detailed instructions is stale. You need a detailed description of what you want Turkers to code for along with examples of each class:

• examples of users that are not relevant (and different reasons why we don't want them)
• positive examples of a few bloggers across types
• examples and advice regarding corner cases You should probably inline example tweets from these users rather than link the the user pages. It helps to pick real examples, possibly from the annotation run we went through.

2) I would group questions 1 and 2 into a single question:

"[a href='link_to_user_page'] User page [/a] -- Classify user's Twitter account:"

• User is missing, private, or banned
• Primary language is not English
• Account is primarily for personal use
• Account is used for promoting online presence or career

We may want to split the last choice into two (online presence/career), but I am worried that it will be even more difficult to classify.

3) I would rephrase the third question (now question 2) as:

"Mark all categories this user tweets about using the checkboxes. Also, select the category they are most focused on using the radio buttons. If you are unsure, do your best. If selecting Other, describe category using the text field."

I would place all choices in a single-column table with checkboxes to the left, radio buttons to the right, and a header describing each one.

Let me know if you have questions.

[bellecarrell]

Current HIT draft: title: Identifying self-promoting twitter accounts link: https://workersandbox.mturk.com/requesters/A2HYFRV03M389R/projects?ref=w_pl_prvw

[bellecarrell]

@abenton i was thinking.. should we make the second question "Mark all categories they --include in their user description--? as you pointed out, tweeting might be part of the strategy more than a marker of what their brand is. also, we annotated based on description only

[bellecarrell]

• [x] grey out style instead of hide

• [x] add inline examples of each type of user

• [x] add inline examples of tweets/descriptions for each category

• [x] modify instructions for 2* (maybe)

• [ ] go through hits in worker sandbox to estimate time spent (probably .6-.8 cents/hit)

[abenton]

..."should we make the second question "Mark all categories they --include in their user description--?"

I like the idea of asking Turkers to use all information at their disposal to categorize users: profile image, name, description, and tweets. I thought we annotated based on user description because it was easier than visiting each page and making a harder decision. Let me know what you think.

[bellecarrell]

@abenton yes, i think that makes sense. i'll update the instructions accordingly

[bellecarrell]

currently debugging greying out + using iframes (just copy/pasted code from @abenton's example code for greying out, internet for iframe).

[bellecarrell]

need to add examples to run through as well as users for each example (just popped in iframes)

[bellecarrell]

@abenton there are 3 HITs you should be able to access in the worker sandbox.

[bellecarrell]

@abenton edited. done for now; fully test in pre-study, but done wrt this issue