search

TurkServer / turkserver-meteor

Web-based, real-time behavioral studies and experiments using Meteor
http://turkserver.readthedocs.io
MIT License
90 stars 23 forks source link

Occasional HIT submission issues. #28

Open mizzao opened 9 years ago

mizzao commented 9 years ago

Seems like turkSubmitTo is sometimes not set and the following error appears:

Oops, looks like there's no route on the client or the server for url: "https://tarsonis.eecs.harvard.edu/mturk/externalSubmit.

@ldworkin found this bug earlier. Not a deal breaker because people can just reload and submit. It seems like it can be fixed by just setting our own submission URL in these rare cases.

Other sources:

mizzao commented 8 years ago

In a related error, Nasrin reports that a you are not logged in [403] error appears occasionally on submission.

I surmise this could be a race condition between hitting the TurkServer submit and getting logged out (and redirected), versus the MTurk form submit completing. To avoid this we could potentially delay the user logout for a bit.

https://github.com/TurkServer/turkserver-meteor/blob/e6650fde1efd07a795cfe21b738c381da840eb9e/server/connections.coffee#L245

mizzao commented 8 years ago

@josephcc, this issue is one potential cause of the user you saw today having a completed assignment that wasn't refresh-able.

I have a potential fix for this based on the link above. Would you be interested in testing it?

josephcc commented 8 years ago

Sure, is there a branch I can pull from?

mizzao commented 8 years ago

Yes, please give this a try: https://github.com/TurkServer/turkserver-meteor/tree/hit-submission-fix

If you submit a HIT 20 times and don't see an error, then this is probably good.

nasrinm commented 8 years ago

I tested the hit-submission-fix branch and it's unreliable. 6/6 times I tested it on sandbox, the first user that gets connected gets the error

Oops, looks like there's no route on the client or the server for url

after hitting submit. Refreshing the page will auto submit, but that's a quite an unpleasant experience for the workers.

mizzao commented 8 years ago

I don't exactly understand your comment. So one of the users gets the error, but not the other?

This is in the sandbox, and not the local testing mode (where you'd definitely see the error), right? And you are using two different browsers...

nasrinm commented 8 years ago

Yes, one of the users gets the error which happens to be the first connected user. This is in sandbox. Submitting the hit locally always gets redirected to the same error in the background but immediately gives you the option of logging in again with a new pop up window in the foreground, I may have seen the same thing testing other versions of turkserver locally but hadn't noticed it as a problem. Yes, I have tested different browsers. Would you or Kevin mind testing the branch on sandbox on your end as well?

nasrinm commented 8 years ago

Another error (separate from the one I reported above) is the following

You have been disconnected from the server. Please check your Internet connection.

error2 capture9

I have not been able to reproduce this systematically, it has happened twice after the hit is submitted with this current branch (first Figure, this is local run), and reported many times happening in the middle of the conversation or while in turkserver portal (second Figure) with the master branch when I tested it on mturk. Obviously the user is not disconnected from the internet when this error shows up.

mizzao commented 8 years ago

@nasrinm I think it's normal for this to happen upon HIT submission, as the Meteor client disconnects and tries to POST to mechanical Turk.

If it's happening in the middle of a task, your server might be getting overloaded or otherwise experiencing some technical issues. Could you start a separate issue for this?

nasrinm commented 8 years ago

Ok that's good if it's normal for this to happen locally. I have added auto-scale to my own Azure server and will see if this will happen at large scale run again (haven't got this error while testing on Sandbox). So l can start the separate issue if it happened on Sandbox/further mturk testing.

mizzao commented 8 years ago

If you had more than a couple of hundred users connected, you will probably overload the CPU of one Node instance and start seeing this (and users will see it as well). I wouldn't post that many HITs at once.

I don't think auto-scaling on Azure will help here, because the app is not set up to use multiple Node instances. (If you did somehow set it up to use multiple Node instances on one DB, TurkServer will not work properly, see #71.)

nasrinm commented 8 years ago

Oh Ok good to know that Turkserver cannot use multiple Node instances. Having more HITs up at the same time helps decrease the wait time for being paired (which the workers dislike very much), which is why I tried having multiple HITs up once.

nasrinm commented 8 years ago

Yes, one of the users gets the error which happens to be the first connected user. This is in sandbox. Submitting the hit locally always gets redirected to the same error in the background but immediately gives you the option of logging in again with a new pop up window in the foreground, I may have seen the same thing testing other versions of turkserver locally but hadn't noticed it as a problem. Yes, I have tested different browsers. Would you or Kevin mind testing the branch on sandbox on your end as well? @nasrinm

Any opinions on this? Do you recommend using Master branch for now?

mizzao commented 8 years ago

Oh Ok good to know that Turkserver cannot use multiple Node instances. Having more HITs up at the same time helps decrease the wait time for being paired (which the workers dislike very much), which is why I tried having multiple HITs up once.

I don't think having 50 HITs available vs 500 HITs will reduce time to get paired. As long as there are a handful of HITs available to accept, there should be the same wait time. The problem is that posting a lot of HITs at the same time will spike the server load and cause other problems.

Sorry, I'm not entirely sure how to fix the submission issues right now. Maybe you should just try putting a message at the bottom of your exit survey that if there is a submission problem, workers should go to HITs Assigned To You and reload this HIT, and it will submit automatically. That should reduce the number of e-mails you receive.

nasrinm commented 8 years ago

My experience is that the attention that 10X available hits for a batch gets is more than 10 times the attention X available hits get and hence more workers will come to be paired and so less wait time. In my other crowdsourcing tasks I have experienced this as well, where I've been able to collect 10times more data in the same amount of time by having a 10X larger batch up. But of course we need the server to work and I am not going to have more than a few HITs up with turkserver, which is OK.

That's actually what I've already been instructing the workers to do, and they do it often, but in general an app that errors is not providing a pleasant experience (payment wise or hits-done-record wise they don't feel good when they see their submission wasn't successful, and I've had cases in which going to the HITs-assigned and refreshing wasn't successful for a few attempts) and we tend to get low ratings as a result. We can choose to be OK with this problem for now though if that's the only option.