search

StanfordHCI / bang

💥 Helping people meet for the first time, more than once 💥
MIT License
2 stars 1 forks source link

Require MTurkers to finish surveys #390

Closed tawnzz closed 5 years ago

tawnzz commented 5 years ago

Hello!

A big problem we're getting is that some users are not finishing surveys (AFKing) during the rounds.

We recently added survey language to deter them from finishing their surveys, but we should add logic that would make sure they do.

See PR #389

i-gao commented 5 years ago

One quick thing that might help is incentivizing participation in each section of the batch.

So say a batch has 3 rounds with 4 surveys (3 mid and 1 final). This is a total of 7 screens. We can divide the expected compensation ($12) / 7 = $1.71.

On each screen, could we place in the upper right the amount participation on "this screen" is worth? A constant string in the corner could just say, "Participation on this section earns $1.71 of your bonus."

Whether or not we actually follow through with not paying them unless they participate is a different question. I'm inclined we could just put the text there as incentive and not follow through.

Thoughts?

tawnzz commented 5 years ago

@i-gao Yup! I had the same thoughts. We recently changed the text to say that they were required to finish the surveys, but I think placing the "worth" of the task is a brilliant idea.

tawnzz commented 5 years ago

Another incentive is putting something like "X Percent of our participations complete the surveys," which would be a form of social pressure. Let's chat more!

deliveryweb commented 5 years ago

I can add special check to final survey; if user has empty mid/pre round survey - block bonus.

markwhiting commented 5 years ago

@deliveryweb sounds good! But I'd like it to not surprise them. We could show them they're done with a survey, so they don't accidentally forget one item.

e.g., an area that shows Survey not complete which changes to Survey complete once it's full.

deliveryweb commented 5 years ago

But they can't submit these surveys. Right?

deliveryweb commented 5 years ago

Okay, how I see it: we have 3 types of activities: pre round survey round (task and chat) mid round survey we count it and we count empty (from user) activities if no survey - it's empty if less than 2 messages in chat - it's empty and we cut bonus with these values For example: 4 rounds, each one has pre,mid survey and chat/task. 12 activities. Batch bonus = 15$. User has messages in every chat, and has 4 surveys. We cut 12 - (4+4)/12 * 15$ =5$ from his bonus. Correct?

markwhiting commented 5 years ago

Perhaps to simplify, lets just tell workers who haven't completed the survey that they will receive a lower bonus for that.

e.g., if there's a survey with 5 questions and I answer 0-4 of them, I see a thing somewhere (bottom of page or in header or something) that says 'You will be bonuses for this survey if it is completed', and once its completed for all 5 questions, that message changes to 'You've completed this survey, please check your answers until the time is up'

Or something like that

Actually changing the bonus is not that important. Its just important that people care about finishing.