As an experiment designer, I want to know if something about the experiment approach is making paid Mechanical Turk participants unhappy, so that I can improve the UX design, compensation strategy, or communication.
As a research manager, I want to be aware of any problems that participants are likely to be discussing on the MTurk Forums, etc. so that we can preserve a good reputation among MTurk workers and ensure long-term access to a high-quality participant pool.
As a scientist, I want to know if any part of the experience that isn't directly measured by the science objectives would lead me to doubt the validity of my data, and I want to have confidence that participants are interacting with the platform in good faith and in the manner that supports the research.
Potential solutions
One way to measure these is with a participant survey that participants can submit after they complete the experiment. An ideal QC survey would draw on best practices from website design, customer satisfaction, and human subjects experiments in either medical or social science. It might measure things like:
any friction or pain points that participants experienced using the UI
the perceived fairness of payment
overall enjoyment of participation
the clarity of instructions and ease of participation
etc.
The survey should:
[x] include both open-response items that can help us troubleshoot problems in our system
[x] include ordinal or numeric items that can be integrated into a dashboard and compared over time
[x] take less than 2-3 minutes to complete
[x] adhere to best practices for academic survey design
We are interested not only in the values of individual responses, but also in aggregations, variances, and behaviors over time, as each of these would give a different understanding of what is happening in the experiment.
Description
Potential solutions
One way to measure these is with a participant survey that participants can submit after they complete the experiment. An ideal QC survey would draw on best practices from website design, customer satisfaction, and human subjects experiments in either medical or social science. It might measure things like:
The survey should:
We are interested not only in the values of individual responses, but also in aggregations, variances, and behaviors over time, as each of these would give a different understanding of what is happening in the experiment.
Related
This issue is complete when