search

TurkServer / turkserver-meteor

Web-based, real-time behavioral studies and experiments using Meteor
http://turkserver.readthedocs.io
MIT License
90 stars 23 forks source link

Handling idle users from admin interface #48

Open ejolly opened 9 years ago

ejolly commented 9 years ago

This might be a feature request, but just wanted to see if it was hidden somewhere. Had an issue where a user submitted an assignment after I forced the experiment instance to end through the admin interface , but turkserver still shows them as connected. Is there a way to disconnect them?

Related, it would also be nice to be able to disconnect a user who has been sitting in an experiment without completing it for a significant amount of time. Is there a suggestion for how to do this, even programmatically?

mizzao commented 9 years ago

This might be a feature request, but just wanted to see if it was hidden somewhere. Had an issue where a user submitted an assignment after I forced the experiment instance to end through the admin interface , but turkserver still shows them as connected. Is there a way to disconnect them?

You might be talking about mizzao/meteor-user-status#29. That's relatively rare though; what's more common is that the user has either (accidentally or deliberately) opened two tabs to the task, and that's why they're still connected. The Maintenance buttons at the top of the Connections and Active Assignments sections can help with this.

Related, it would also be nice to be able to disconnect a user who has been sitting in an experiment without completing it for a significant amount of time. Is there a suggestion for how to do this, even programmatically?

I think you can force a user offline programmatically, but their browser will just disconnect. Usually I just use the HIT timeout to accomplish this, then use the Cancel Assignments button to set them as returned and clear the user's connection state.

ejolly commented 9 years ago
what's more common is that the user has either (accidentally or deliberately) opened two tabs to the task, and that's why they're still connected. The Maintenance buttons at the top of the Connections and Active Assignments sections can help with this.

Ok this seems pretty likely because they eventually disconnected themselves after some time.

I think you can force a user offline programmatically, but their browser will just disconnect.

Is there turkserver code example for this somewhere?

Usually I just use the HIT timeout to accomplish this, then use the Cancel Assignments button to set them as returned and clear the user's connection state.

Sorry here you mean the Assignment duration when creating a HIT type? If the HIT duration runs out does it just automatically get returned? My concern is that if one user in a two (or more) user experiment causes the HIT duration to run out, then it isn't fair for the second user to not earn any money. I think it would be preferable to see if a single user has timed out, if so disconnect them, then push the remaining user to the exit survey. This something like that currently possible?

mizzao commented 9 years ago

Is there turkserver code example for this somewhere?

No, but you can use standard Meteor examples of logging a user out.

Sorry here you mean the Assignment duration when creating a HIT type? If the HIT duration runs out does it just automatically get returned? My concern is that if one user in a two (or more) user experiment causes the HIT duration to run out, then it isn't fair for the second user to not earn any money. I think it would be preferable to see if a single user has timed out, if so disconnect them, then push the remaining user to the exit survey. This something like that currently possible?

  • HIT Lifetime = how long the HIT is available to be taken before it expires.
  • Assignment Duration = how long a user has to do a HIT after accepting it.

You want to set the assignment duration to be a bit longer than what your study takes so that users have some flexibility if there are submission problems and they don't feel rushed. However, you also don't want to set this too long so that if you know the assignment has expired you can assume the user isn't coming back and set it as returned. I usually do the latter at the end of a day, etc.

You can definitely push users out of instances into the lobby (and then the exit survey); see the cooperation experiment for an example.

ejolly commented 9 years ago

Ok relatedly, (but perhaps a separate issue), is there currently functionality in Turkserver so that users can return a HIT from within a Meteor App? I.e. something like the submit HIT functionality. Example use case might be if they don't pass a comprehension quiz prior to starting an experiment.

mizzao commented 9 years ago

Users return a HIT by pushing the Return HIT button directly in MTurk. When they do this they will just go offline. This is hard to tell apart from just a temporary disconnection, but you know they've returned it if it's been longer than the assignment duration. That's what those "Maintenance" buttons are for. We could improve this behavior to immediately detect returns by setting up the Notification Receptor APIs, but those have largely seemed not worthwhile for the amount of fidgety-ness required.

You can force a user to return a HIT by displaying some message that basically tells them to do so and prevents them from proceeding further. In this case it may be helpful to immediately record a user's session as returned in the database, after they disconnect.