Task Redundancies

[adikeinan]

Even though the redundancy is now set to 2, we get many more tasks per card.

Tasks from the first line are from a few days ago. This happens everywhere...

[adikeinan]

The two Arreton Hoard applications are free of this bug.

[portableant]

This does account for old task runs though prior to the change. Arreton has always been set at 2.

[adikeinan]

But it seems like there are new tasks per cards that already had a few different transcriptions. The first one on the screenshot, for example, already had 4 transcribers, and a few days ago (30 May) it was presented again and transcribed again..

[ahb108]

Hmmm, really? My impression is that this is working fine, but maybe not. Fortunately B8 is nearly done, so we can monitor the others for how they behave?

[adikeinan]

Look for example at this json of task runs for task #6260 of the photomasking application: http://crowdsourced.micropasts.org/app/photomasking/6260/results.json 8 different people did this task, the last one was on 3 June (much after we allegedly fixed this issue). Instead of presenting users with objects that were masked less than 3 times, the application keeps bringing up images that had more than enough masking tasks… It’s a shame, we’re going to have too many results, and it slows down the completion of each application.

[ahb108]

Worrying. I have just asked @teleyinex about this...and have changes the apps to Breadth First task allocation for now.

[teleyinex]

Hi there,

You have selected a scheduler (here you can see it: http://crowdsourced.micropasts.org/api/app/16) , Breadth First that does not respect the n_answers value. It's in the documentation, please, check it carefully before selecting or modifying the default scheduler (I copy and paste the docs):

The Breadth First scheduler has the following features:

• It sends the tasks in the order that were created, first in first out.
• It ignores the Task Redundancy value, so it will keep sending tasks no matter even though that value has been achieved.
• It sends always the task with the least number of task runs in the system.
• A task will be never marked as completed, as the Task Redundancy is not respected.

In summary, from the point of view of a user (authenticated or anonymous) the system will be sending the project’s tasks that have less answers (in case of not having an answer, the creation time will be used to send them like in a FIFO –first in first out).

From the point of view of the project, the scheduler will be trying to obtain as soon as possible an answer for all the available tasks.

Therefore, the solution is to use the default scheduler.

[portableant]

So we should RTFM.

[ahb108]

Err no, it's a bug (see our email conversation) and https://github.com/PyBossa/pybossa/issues/823

[portableant]

Right, so now I am confused. Is this a bug or is this our fault?

[portableant]

Never mind, found your email conversation now. Emails getting filtered to spam.

[teleyinex]

And this is solved here: https://github.com/PyBossa/pybossa/pull/824 We'll merge it tomorrow morning or so, so you can actually copy and paste that code in your own version. Do not upgrade yet your system as you'll need to do lots of changes ;-)

[portableant]

Okay, just let me know when it is merged and I'll paste code in. Thanks!

[teleyinex]

Hi,

You can paste it now, as it is passing the unit tests. However, we do check between my us (the team) the code, just in case something is broken or forgotten ;-)

This will solve the problem for you in case you need to change again redundancy, as now it will update properly the status.

Cheers,

Daniel

[ahb108]

Great! @portableant can you paste this when convenient and then update the schedulers to Default by any chance?

[adikeinan]

Hi @portableant, did you make the code change? This redundancy is still going on, at least on the photomasking application, e.g. tasks from Friday 13 June for photos that have already been masked 8 times..

[portableant]

Yes.

[adikeinan]

Sorry for keep bringing bad news... This happened for the first time last week for the Arreton Hoard photomasking. You can see that one photo was masked 4 times: http://crowdsourced.micropasts.org/app/photomaskingArreton/tasks/browse

[portableant]

Can you liaise with Daniel? I've got too much on this week to do this.

[adikeinan]

Sure, will do.

[teleyinex]

Hi,

This is happening because the scheduler works as follows. Imagine you have only one task left to be completed. Volunteer A arrives and asks for a task, the server delivers the last task, ok? While volunteer A is solving the task, a new volunteer arrives, volunteer B and ask for a task in the same application. PyBossa then sends the same task, the last one, because volunteer A has not posted yet an answer and it does not know if this user will send an answer or not (maybe the user closes the browser, leaves the desk, etc.). This behavior is done on purpose because we want to assure that an application gets all the answers for the projects, and it's better to get an extra task than locking a task for a period of time (this could be even used by bad users to lock tasks for longer periods).

Thus, I guess you are experiencing this "problem" I would not call it a problem, but a feature to be safe. From the point of view of the users, they don't see anything wrong, and the truth is that when they requested a task, there was one left so they get it.

I hope this helps!