Connect RESCOP Service to validate annotation quality

[tmbo]

To evaluate a tracers performance we are going to use a RESCOP service that given a tracings data evaluate the fitness of that tracing.

Log Time

[manuel-berning]

I just wanted to follow up on my mail from yesterday here. I am still re-implementing the RESCOP code once written by MH currently. After that I would need some knowledge of how the interface should be designed (what would you need from my side (e.g. URL/port?), what will you send? how will you sent it?).

[tmbo]

Depends, in which language are you implementing the service?

[manuel-berning]

I thought that is exactly what should not matter for a service? We just need a definition of an API to exchange data between wK and RESCOP and then either could we written in tab-whitespace for all we care?

All the RESCOP logic is in matlab, it is not a problem to start matlab from any other language and retrieve the results. We could compile the MATLAB code if we want to run it on a machine without access to a license server.

So I think I could implement it in whatever language we like (please do not chose tab-whitespace xD ). I rather meant in terms of what would define the API? Or is there a more recent concept I missed?

[boergens]

@tmbo do you have a preference whether you want to send the data as multipart/form-data or application/x-www-form-urlencoded The data will be small (just a few nodes and edges) I assume the request will be made from the frontend, right?

[tmbo]

Sure, it doesn't matter. But still, I was interested :D

It should be a service that accepts a POST my.prescop.com/api/services/score-nml with the body of the POST beeing a properly formatted NML. The response should (preferably but not necessarily) be JSON of some sort with some result.

[tmbo]

Content type will be either text/text or application/xml (depends if the nml is proper xml, don't remember)

[manuel-berning]

Could we establish a method to also get the full webKNOSSOS taskID?

e.g. post to POST my.prescop.com/api/services/score-nml/(taskID)

I think this would be beneficial for us for better logging and/or debugging.

[tmbo]

sure, I can append the taskid as a parameter, e.g. ?taskId=xyz123

[manuel-berning]

Sounds good!

[boergens]

@tmbo Should I take this over? I have some time now and it only seems front-end related

[tmbo]

I don't think this frontend related. The annotation quality doesn't need to be validated in the frontend (furthermore we don't have any tools for nml generation there). The process is comply asynchronous. MTurk worker works. Finishes. Done. At some point we check his annotation and decide if he gehts rewarded.

Btw. as far as I understood, payment should NOT be bound to quality of work. As long as a worker sticks to the given instructions and there is no violation the worker should be paid.

Nevertheless, we can use the evaluation for e.g. qualification assignment.

[boergens]

No, that's not the plan. The scheme is that after 2um of annotation the data is sent on the fly to the RESCOP server. If the answer is negative tracing is stopped and the annotator gets a window: "Sorry, your performance is insufficient, you can't continue" If the answer is positive the annotator can finish the whole 10min (and then indeed will get paid even if they made a mistake)

[tmbo]

It does seem to complicate things, what is the benefit of having that step? Are you afraid the workers can't cope with the HIT? In that case instructions should be improved.

[boergens]

we see that only about have of annotators "get" it after the video.. Any suggestions how to improve the video are always most welcome

[manuel-berning]

have = half

[boergens]

:see_no_evil:

[tmbo]

I really don't think this is a good idea. Even if half the people don't get it (and if thats the case I think there is quite some room for improvement in the instructions or tracing view and we should work on that instead of kicking out people) why not let them trace? Maybe they learn something on the way and will get to be good tracers in the end. We can still punish them after they are done and tell them that they tracing is bad / malicious / ...

Maybe the people who don't get it can even give valuable feedback. For the evaluation you can still throw them out of your analysis.

[manuel-berning]

I do not think it is an instruction problem. Some people for example just keep flying straight the whole way (maybe they do not care or sth?). But if you think you can get them all, please work on the movie and let us know when you found a version that works for everybody. Why not let them trace? -> Because somebody has to pay for it (the problem is not paying for this test but estimating what it would take in real use case).

[tmbo]

Don't get me wrong, I don not want to keep everyone as a tracer. We got the great possibility to ban tracers from taking new assignments. But I just don't see the point of doing that in the middle of an assignment because:

• Way harder to implement (needs quite some front end changes, backend as well)
• Does not save money. We can still reject the work afterwards and we won't pay a cent more, there is no benefit in doing it in the middle of the assignment
• Checking the quality in the backend is simpler and more secure

And remember, the crowd that is working at mturk needs to keep their so called reputation score up, because every good paying job looks at the percentage of faulty assignments the worker made. Hence, I think the percentage of people maliciously trying to cheat is low.

Getting back to your last statement "the problem is not paying for this test but estimating what it would take in real use case": Keeping a side that the backend solution is actually cheaper. I, again, strongly disagree. Right know the priority is the paper, right? We are running on a tight schedule, so if there is no benefit in having this more involved implementation at the moment leet's keep focused on what is next.

Thats it from my side (and the longest mail I have written for quite some time :D), if I still didn't convince anyone that this is a bad idea, I am willing to adjust my views according to the majority ones.

Best, Tom

Tom Bocklisch

mobile: +49 176 4206 1357 web: http://scm.io http://www.scalableminds.com/ http://scm.io/

scalable minds UG (haftungsbeschränkt) & Co. KG Stahnsdorfer Straße 152a | 14482 Potsdam

Handelsregister: HRA 5753 Registergericht Potsdam

Geschäftsführer: Tom Bocklisch, Tom Herold, Norman Rzepka, Thomas Werkmeister

Am 18. August 2016 um 17:51:33, Manuel Berning (notifications@github.com) schrieb:

I do not think it is an instruction problem. Some people for example just keep flying straight the whole way (maybe they do not care or sth?). But if you think you can get them all, please work on the movie and let us know when you found a version that works for everybody. Why not let them trace? -> Because somebody has to pay for it (the problem is not paying for this test but estimating what it would take in real use case).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/scalableminds/oxalis/issues/1440#issuecomment-240767683, or mute the thread https://github.com/notifications/unsubscribe-auth/ABDCrP4mhJ6c6XNTSbIwx_GzNpsFsedRks5qhH9-gaJpZM4Jlzl7 .

[manuel-berning]

I see your point of course & i like simple solutions!

We have of course discussed this point very extensively in the last call (among others) to a large degree. I think that would mean decreasing the length of the assignment from ~15 min to 2-3 min? (among other things)? Also we usually discussed it under the assumption that most tracers will not stay with the task/service for a long time (and thus keeping users between assignments is difficult if you do not overpay). This also means that we have to make sure that we do not simply pay them to watch the introductory movie (~3min)?

To your point to my last point, the goal here is to make a statement such as: "Using micro-work on amazon mTurk combined with webKnossos we show here that it is possible to reconstruct neurites from SBEM datasets in the cortex at a cost of X Dollar/Euro per mm path length" If we want to make such a statement we need to show it as well? We believe that would be the main number of interest for potential users/customers. Of course we can always also pay for work we cannot use for reconstruction, but this will obviously make this number worse (could be up to half if no user comes back)?

Agreed that paper has priority.

[tmbo]

Remember, the number of paid dollars does not go up if we validate afterwards (reject work & don’t pay => no money lost)! So that number will look the same for the paper.

Reading through the docs I think 15min is reasonable. I don’t know how large the percentage of people is that do more than one task, we will see that after the first experiments. If they don’t trace and just watch the video this is an easy reject for us, hence no payment for that user.

I am not sure if we are on the same page about the mturk workflow:

1. User looks at assignment
2. User takes assignment
3. User works on assignment
4. User finishes assignment
5. We have a defined amount of time to evaluate his work (depends on us, maximum 30 days). If we don’t reject his work in that interval it will get accepted.
6. Only if we accept (and only after we have done so) the user gets paid

Best, Tom

Tom Bocklisch

mobile: +49 176 4206 1357 web: http://scm.io http://www.scalableminds.com/ http://scm.io/

scalable minds UG (haftungsbeschränkt) & Co. KG Stahnsdorfer Straße 152a | 14482 Potsdam

Handelsregister: HRA 5753 Registergericht Potsdam

Geschäftsführer: Tom Bocklisch, Tom Herold, Norman Rzepka, Thomas Werkmeister

Am 18. August 2016 um 21:51:53, Manuel Berning (notifications@github.com) schrieb:

I see your point of course & i like simple solutions!

We have of course discussed this point very extensively in the last call (among others) to a large degree. I think that would mean decreasing the length of the assignment from ~15 min to 2-3 min? (among other things)? Also we usually discussed it under the assumption that most tracers will not stay with the task/service for a long time (and thus keeping users between assignments is difficult if you do not overpay). This also means that we have to make sure that we do not simply pay them to watch the introductory movie (~3min)?

To your point to my last point, the goal here is to make a statement such as: "Using micro-work on amazon mTurk combined with webKnossos we show here that it is possible to reconstruct neurites from SBEM datasets in the cortex at a cost of X Dollar/Euro per mm path length" If we want to make such a statement we need to show it as well? We believe that would be the main number of interest for potential users/customers. Of course we can always also pay for work we cannot use for reconstruction, but this will obviously make this number worse (could be up to half if no user comes back)?

Agreed that paper has priority.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/scalableminds/oxalis/issues/1440#issuecomment-240836424, or mute the thread https://github.com/notifications/unsubscribe-auth/ABDCrNXXv4SALodFJ_fVgDKQfTP_6AGVks5qhLfPgaJpZM4Jlzl7 .

[tmbo]

Hi Tom,

thanks for your mail! Lively discussion is always appreciated :-)

On 18 August 2016 at 16:28, Tom Bocklisch tom.bocklisch@scalableminds.com wrote:

Don't get me wrong, I don not want to keep everyone as a tracer. We got the great possibility to ban tracers from taking new assignments. But I just don't see the point of doing that in the middle of an assignment because:

Actually I'm not super keen on banning. If you have a robust payment filter there is nothing wrong with encouraging to give it another go (they watch the video again and perhaps then the "coin drops")

• Way harder to implement (needs quite some front end changes, backend as well)

I'm not convinced it would take backend changes. I think Manuel is a big boy and can live with node data as JSON instead of XML. I'm happy to do the frontend changes.

• Does not save money. We can still reject the work afterwards and we won't pay a cent more, there is no benefit in doing it in the middle of the assignment

Actually there is. So the way it works is that tasks are layered with slightly overlapping bounding boxes. The "ground truth" of the new task is the consensus of the old. We need this filter, otherwise our consensus building doesn't work. So we know after 2um whether we will look at a result at all. And still we would let the person continue for another 8 minutes without the slightest chance of payment? That sounds cruel. Also, for new annotators it is shown what our payout fraction is. So if someone quits the tasks after two minutes, that is much better for us than if they submit and we don't pay them

• Checking the quality in the backend is simpler and more secure

I thought about that. It is of course conceivable that someone would find a way to ignore the Result coming back from RESCOPaaS. But we would still save the result and then we could still block the payment for those cases (we would have to do that anyway in case they just wait it out and press submit after 10mins.

And remember, the crowd that is working at mturk needs to keep their so called reputation score up, because every good paying job looks at the percentage of faulty assignments the worker made. Hence, I think the percentage of people maliciously trying to cheat is low.

That's exactly my point. If somebody wants to try again, they are welcome to and we should treat them with dignity and let them know as soon as we know that there is no chance of a payout.

Getting back to your last statement "the problem is not paying for this test but estimating what it would take in real use case": Keeping a side that the backend solution is actually cheaper. I, again, strongly disagree. Right know the priority is the paper, right? We are running on a tight schedule, so if there is no benefit in having this more involved implementation at the moment leet's keep focused on what is next.

Thats it from my side (and the longest mail I have written for quite some time :D), if I still didn't convince anyone that this is a bad idea, I am willing to adjust my views according to the majority ones.

I'm also trying the same, this way we can hope to arrive at the solution that is actually best

Best wishes,

Kevin

PS: I saw that you and Manuel went through another iteration of mails in the meantime but I feel what I wrote still has relevance

Best, Tom

Tom Bocklisch

mobile: +49 176 4206 1357 web: http://scm.io http://www.scalableminds.com/ http://scm.io/

scalable minds UG (haftungsbeschränkt) & Co. KG Stahnsdorfer Straße 152a | 14482 Potsdam

Handelsregister: HRA 5753 Registergericht Potsdam

Geschäftsführer: Tom Bocklisch, Tom Herold, Norman Rzepka, Thomas Werkmeister

Am 18. August 2016 um 17:51:33, Manuel Berning (notifications@github.com) schrieb:

I do not think it is an instruction problem. Some people for example just keep flying straight the whole way (maybe they do not care or sth?). But if you think you can get them all, please work on the movie and let us know when you found a version that works for everybody. Why not let them trace? -> Because somebody has to pay for it (the problem is not paying for this test but estimating what it would take in real use case).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/scalableminds/oxalis/issues/1440#issuecomment-240767683, or mute the thread https://github.com/notifications/unsubscribe-auth/ABDCrP4mhJ6c6XNTSbIwx_GzNpsFsedRks5qhH9-gaJpZM4Jlzl7 .

[tmbo]

Hi guys

Let's discuss today at 2pm if possible please. (All aspects, including the one below)

Best,

m

Moritz Helmstaedter

Department of Connectomics Max Planck Institute for Brain Research Max-von-Laue Str. 4, 60316 Frankfurt, Germany

www.brain.mpg.de/helmstaedterhttp://www.brain.mpg.de/helmstaedter

Department office: mhoffice@brain.mpg.demailto:mhoffice@brain.mpg.de (ph) +49 69 850033 3001tel:+49%2069%20850033%203001

On 18 Aug 2016, at 22:12, Kevin Boergens kevin.boergens@brain.mpg.de<mailto:kevin.boergens@brain.mpg.de> wrote:

Hi Tom,

thanks for your mail! Lively discussion is always appreciated :-)

On 18 August 2016 at 16:28, Tom Bocklisch tom.bocklisch@scalableminds.com<mailto:tom.bocklisch@scalableminds.com> wrote: Don't get me wrong, I don not want to keep everyone as a tracer. We got the great possibility to ban tracers from taking new assignments. But I just don't see the point of doing that in the middle of an assignment because: Actually I'm not super keen on banning. If you have a robust payment filter there is nothing wrong with encouraging to give it another go (they watch the video again and perhaps then the "coin drops")

• Way harder to implement (needs quite some front end changes, backend as well) I'm not convinced it would take backend changes. I think Manuel is a big boy and can live with node data as JSON instead of XML. I'm happy to do the frontend changes.
• Does not save money. We can still reject the work afterwards and we won't pay a cent more, there is no benefit in doing it in the middle of the assignment Actually there is. So the way it works is that tasks are layered with slightly overlapping bounding boxes. The "ground truth" of the new task is the consensus of the old. We need this filter, otherwise our consensus building doesn't work. So we know after 2um whether we will look at a result at all. And still we would let the person continue for another 8 minutes without the slightest chance of payment? That sounds cruel. Also, for new annotators it is shown what our payout fraction is. So if someone quits the tasks after two minutes, that is much better for us than if they submit and we don't pay them
• Checking the quality in the backend is simpler and more secure I thought about that. It is of course conceivable that someone would find a way to ignore the Result coming back from RESCOPaaS. But we would still save the result and then we could still block the payment for those cases (we would have to do that anyway in case they just wait it out and press submit after 10mins.

And remember, the crowd that is working at mturk needs to keep their so called reputation score up, because every good paying job looks at the percentage of faulty assignments the worker made. Hence, I think the percentage of people maliciously trying to cheat is low. That's exactly my point. If somebody wants to try again, they are welcome to and we should treat them with dignity and let them know as soon as we know that there is no chance of a payout.

Getting back to your last statement "the problem is not paying for this test but estimating what it would take in real use case": Keeping a side that the backend solution is actually cheaper. I, again, strongly disagree. Right know the priority is the paper, right? We are running on a tight schedule, so if there is no benefit in having this more involved implementation at the moment leet's keep focused on what is next.

Thats it from my side (and the longest mail I have written for quite some time :D), if I still didn't convince anyone that this is a bad idea, I am willing to adjust my views according to the majority ones. I'm also trying the same, this way we can hope to arrive at the solution that is actually best

Best wishes,

Kevin

PS: I saw that you and Manuel went through another iteration of mails in the meantime but I feel what I wrote still has relevance

Best, Tom

Tom Bocklisch

mobile: +49 176 4206 1357 web: http://scm.iohttp://www.scalableminds.com/ http://scm.io/

scalable minds UG (haftungsbeschränkt) & Co. KG Stahnsdorfer Straße 152a | 14482 Potsdam

Handelsregister: HRA 5753 Registergericht Potsdam

Geschäftsführer: Tom Bocklisch, Tom Herold, Norman Rzepka, Thomas Werkmeister

Am 18. August 2016 um 17:51:33, Manuel Berning (notifications@github.commailto:notifications@github.com) schrieb:

I do not think it is an instruction problem. Some people for example just keep flying straight the whole way (maybe they do not care or sth?). But if you think you can get them all, please work on the movie and let us know when you found a version that works for everybody. Why not let them trace? -> Because somebody has to pay for it (the problem is not paying for this test but estimating what it would take in real use case).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/scalableminds/oxalis/issues/1440#issuecomment-240767683, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABDCrP4mhJ6c6XNTSbIwx_GzNpsFsedRks5qhH9-gaJpZM4Jlzl7.

[tmbo]

Hi,

would earlier work for you too?

On Aug 19, 2016 8:07 AM, "Moritz Helmstaedter" < moritz.helmstaedter@brain.mpg.de> wrote:

Hi guys

Let's discuss today at 2pm if possible please. (All aspects, including the one below)

Best,

m

Moritz Helmstaedter

Department of Connectomics Max Planck Institute for Brain Research

Max-von-Laue Str. 4, 60316 Frankfurt, Germany

www.brain.mpg.de/helmstaedter

Department office: mhoffice@brain.mpg.de (ph) +49 69 850033 3001 <+49%2069%20850033%203001>

On 18 Aug 2016, at 22:12, Kevin Boergens kevin.boergens@brain.mpg.de wrote:

Hi Tom,

thanks for your mail! Lively discussion is always appreciated :-)

On 18 August 2016 at 16:28, Tom Bocklisch <tom.bocklisch@scalableminds.com

wrote:

Don't get me wrong, I don not want to keep everyone as a tracer. We got the great possibility to ban tracers from taking new assignments. But I just don't see the point of doing that in the middle of an assignment because:

Actually I'm not super keen on banning. If you have a robust payment filter there is nothing wrong with encouraging to give it another go (they watch the video again and perhaps then the "coin drops")

• Way harder to implement (needs quite some front end changes, backend as well)

I'm not convinced it would take backend changes. I think Manuel is a big boy and can live with node data as JSON instead of XML. I'm happy to do the frontend changes.

• Does not save money. We can still reject the work afterwards and we won't pay a cent more, there is no benefit in doing it in the middle of the assignment

Actually there is. So the way it works is that tasks are layered with slightly overlapping bounding boxes. The "ground truth" of the new task is the consensus of the old. We need this filter, otherwise our consensus building doesn't work. So we know after 2um whether we will look at a result at all. And still we would let the person continue for another 8 minutes without the slightest chance of payment? That sounds cruel. Also, for new annotators it is shown what our payout fraction is. So if someone quits the tasks after two minutes, that is much better for us than if they submit and we don't pay them

• Checking the quality in the backend is simpler and more secure

I thought about that. It is of course conceivable that someone would find a way to ignore the Result coming back from RESCOPaaS. But we would still save the result and then we could still block the payment for those cases (we would have to do that anyway in case they just wait it out and press submit after 10mins.

And remember, the crowd that is working at mturk needs to keep their so called reputation score up, because every good paying job looks at the percentage of faulty assignments the worker made. Hence, I think the percentage of people maliciously trying to cheat is low.

That's exactly my point. If somebody wants to try again, they are welcome to and we should treat them with dignity and let them know as soon as we know that there is no chance of a payout.

Getting back to your last statement "the problem is not paying for this test but estimating what it would take in real use case": Keeping a side that the backend solution is actually cheaper. I, again, strongly disagree. Right know the priority is the paper, right? We are running on a tight schedule, so if there is no benefit in having this more involved implementation at the moment leet's keep focused on what is next.

Thats it from my side (and the longest mail I have written for quite some time :D), if I still didn't convince anyone that this is a bad idea, I am willing to adjust my views according to the majority ones.

I'm also trying the same, this way we can hope to arrive at the solution that is actually best

Best wishes,

Kevin

PS: I saw that you and Manuel went through another iteration of mails in the meantime but I feel what I wrote still has relevance

Best, Tom

Tom Bocklisch

mobile: +49 176 4206 1357 web: http://scm.io http://www.scalableminds.com/ http://scm.io/

scalable minds UG (haftungsbeschränkt) & Co. KG Stahnsdorfer Straße 152a | 14482 Potsdam

Handelsregister: HRA 5753 Registergericht Potsdam

Geschäftsführer: Tom Bocklisch, Tom Herold, Norman Rzepka, Thomas Werkmeister

Am 18. August 2016 um 17:51:33, Manuel Berning (notifications@github.com) schrieb:

I do not think it is an instruction problem. Some people for example just keep flying straight the whole way (maybe they do not care or sth?). But if you think you can get them all, please work on the movie and let us know when you found a version that works for everybody. Why not let them trace? -> Because somebody has to pay for it (the problem is not paying for this test but estimating what it would take in real use case).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/scalableminds/oxalis/issues/1440#issuecomment-240767683, or mute the thread https://github.com/notifications/unsubscribe-auth/ABDCrP4mhJ6c6XNTSbIwx_GzNpsFsedRks5qhH9-gaJpZM4Jlzl7 .

[tmbo]

Hi,

sorry premature sending. I'm at a doctor's appointment later so 2pm is a bit tight.

Best, Kevin

On Aug 19, 2016 8:07 AM, "Moritz Helmstaedter" < moritz.helmstaedter@brain.mpg.de> wrote:

Hi guys

Let's discuss today at 2pm if possible please. (All aspects, including the one below)

Best,

m

Moritz Helmstaedter

Department of Connectomics Max Planck Institute for Brain Research

Max-von-Laue Str. 4, 60316 Frankfurt, Germany

www.brain.mpg.de/helmstaedter

Department office: mhoffice@brain.mpg.de (ph) +49 69 850033 3001 <+49%2069%20850033%203001>

On 18 Aug 2016, at 22:12, Kevin Boergens kevin.boergens@brain.mpg.de wrote:

Hi Tom,

thanks for your mail! Lively discussion is always appreciated :-)

On 18 August 2016 at 16:28, Tom Bocklisch <tom.bocklisch@scalableminds.com

wrote:

Don't get me wrong, I don not want to keep everyone as a tracer. We got the great possibility to ban tracers from taking new assignments. But I just don't see the point of doing that in the middle of an assignment because:

Actually I'm not super keen on banning. If you have a robust payment filter there is nothing wrong with encouraging to give it another go (they watch the video again and perhaps then the "coin drops")

• Way harder to implement (needs quite some front end changes, backend as well)

I'm not convinced it would take backend changes. I think Manuel is a big boy and can live with node data as JSON instead of XML. I'm happy to do the frontend changes.

• Does not save money. We can still reject the work afterwards and we won't pay a cent more, there is no benefit in doing it in the middle of the assignment

Actually there is. So the way it works is that tasks are layered with slightly overlapping bounding boxes. The "ground truth" of the new task is the consensus of the old. We need this filter, otherwise our consensus building doesn't work. So we know after 2um whether we will look at a result at all. And still we would let the person continue for another 8 minutes without the slightest chance of payment? That sounds cruel. Also, for new annotators it is shown what our payout fraction is. So if someone quits the tasks after two minutes, that is much better for us than if they submit and we don't pay them

• Checking the quality in the backend is simpler and more secure

I thought about that. It is of course conceivable that someone would find a way to ignore the Result coming back from RESCOPaaS. But we would still save the result and then we could still block the payment for those cases (we would have to do that anyway in case they just wait it out and press submit after 10mins.

And remember, the crowd that is working at mturk needs to keep their so called reputation score up, because every good paying job looks at the percentage of faulty assignments the worker made. Hence, I think the percentage of people maliciously trying to cheat is low.

That's exactly my point. If somebody wants to try again, they are welcome to and we should treat them with dignity and let them know as soon as we know that there is no chance of a payout.

Getting back to your last statement "the problem is not paying for this test but estimating what it would take in real use case": Keeping a side that the backend solution is actually cheaper. I, again, strongly disagree. Right know the priority is the paper, right? We are running on a tight schedule, so if there is no benefit in having this more involved implementation at the moment leet's keep focused on what is next.

Thats it from my side (and the longest mail I have written for quite some time :D), if I still didn't convince anyone that this is a bad idea, I am willing to adjust my views according to the majority ones.

I'm also trying the same, this way we can hope to arrive at the solution that is actually best

Best wishes,

Kevin

PS: I saw that you and Manuel went through another iteration of mails in the meantime but I feel what I wrote still has relevance

Best, Tom

Tom Bocklisch

mobile: +49 176 4206 1357 web: http://scm.io http://www.scalableminds.com/ http://scm.io/

scalable minds UG (haftungsbeschränkt) & Co. KG Stahnsdorfer Straße 152a | 14482 Potsdam

Handelsregister: HRA 5753 Registergericht Potsdam

Geschäftsführer: Tom Bocklisch, Tom Herold, Norman Rzepka, Thomas Werkmeister

Am 18. August 2016 um 17:51:33, Manuel Berning (notifications@github.com) schrieb:

I do not think it is an instruction problem. Some people for example just keep flying straight the whole way (maybe they do not care or sth?). But if you think you can get them all, please work on the movie and let us know when you found a version that works for everybody. Why not let them trace? -> Because somebody has to pay for it (the problem is not paying for this test but estimating what it would take in real use case).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/scalableminds/oxalis/issues/1440#issuecomment-240767683, or mute the thread https://github.com/notifications/unsubscribe-auth/ABDCrP4mhJ6c6XNTSbIwx_GzNpsFsedRks5qhH9-gaJpZM4Jlzl7 .

[tmbo]

I can do 1:15p not sure about others?

Moritz Helmstaedter

Department of Connectomics Max Planck Institute for Brain Research Max-von-Laue-Str. 4 D-60438 Frankfurt, Germany

mh@brain.mpg.demailto:mh@brain.mpg.de www.brain.mpg.de/helmstaedterhttp://www.brain.mpg.de/helmstaedter

Department office: mhoffice@brain.mpg.demailto:mhoffice@brain.mpg.de (ph) +49 69 850033 3001

From: Kevin Boergens [mailto:kevin.boergens@brain.mpg.de] Sent: Friday, August 19, 2016 12:26 PM To: Moritz Helmstaedter Cc: Manuel Berning; scalableminds/oxalis; Mention; scalableminds/oxalis; Manuel Berning; Tom Bocklisch Subject: Re: Call at 2pm today Re: [scalableminds/oxalis] Connect RESCOP Service to validate annotation quality (#1440)

Hi,

sorry premature sending. I'm at a doctor's appointment later so 2pm is a bit tight.

Best, Kevin

On Aug 19, 2016 8:07 AM, "Moritz Helmstaedter" moritz.helmstaedter@brain.mpg.de<mailto:moritz.helmstaedter@brain.mpg.de> wrote: Hi guys

Let's discuss today at 2pm if possible please. (All aspects, including the one below)

Best,

m

Moritz Helmstaedter

Department of Connectomics Max Planck Institute for Brain Research Max-von-Laue Str. 4, 60316 Frankfurt, Germany www.brain.mpg.de/helmstaedterhttp://www.brain.mpg.de/helmstaedter Department office: mhoffice@brain.mpg.demailto:mhoffice@brain.mpg.de (ph) +49 69 850033 3001tel:+49%2069%20850033%203001

On 18 Aug 2016, at 22:12, Kevin Boergens kevin.boergens@brain.mpg.de<mailto:kevin.boergens@brain.mpg.de> wrote: Hi Tom,

thanks for your mail! Lively discussion is always appreciated :-)

On 18 August 2016 at 16:28, Tom Bocklisch tom.bocklisch@scalableminds.com<mailto:tom.bocklisch@scalableminds.com> wrote: Don't get me wrong, I don not want to keep everyone as a tracer. We got the great possibility to ban tracers from taking new assignments. But I just don't see the point of doing that in the middle of an assignment because: Actually I'm not super keen on banning. If you have a robust payment filter there is nothing wrong with encouraging to give it another go (they watch the video again and perhaps then the "coin drops")

• Way harder to implement (needs quite some front end changes, backend as well) I'm not convinced it would take backend changes. I think Manuel is a big boy and can live with node data as JSON instead of XML. I'm happy to do the frontend changes.
• Does not save money. We can still reject the work afterwards and we won't pay a cent more, there is no benefit in doing it in the middle of the assignment Actually there is. So the way it works is that tasks are layered with slightly overlapping bounding boxes. The "ground truth" of the new task is the consensus of the old. We need this filter, otherwise our consensus building doesn't work. So we know after 2um whether we will look at a result at all. And still we would let the person continue for another 8 minutes without the slightest chance of payment? That sounds cruel. Also, for new annotators it is shown what our payout fraction is. So if someone quits the tasks after two minutes, that is much better for us than if they submit and we don't pay them
• Checking the quality in the backend is simpler and more secure I thought about that. It is of course conceivable that someone would find a way to ignore the Result coming back from RESCOPaaS. But we would still save the result and then we could still block the payment for those cases (we would have to do that anyway in case they just wait it out and press submit after 10mins.

And remember, the crowd that is working at mturk needs to keep their so called reputation score up, because every good paying job looks at the percentage of faulty assignments the worker made. Hence, I think the percentage of people maliciously trying to cheat is low. That's exactly my point. If somebody wants to try again, they are welcome to and we should treat them with dignity and let them know as soon as we know that there is no chance of a payout.

Getting back to your last statement "the problem is not paying for this test but estimating what it would take in real use case": Keeping a side that the backend solution is actually cheaper. I, again, strongly disagree. Right know the priority is the paper, right? We are running on a tight schedule, so if there is no benefit in having this more involved implementation at the moment leet's keep focused on what is next.

Thats it from my side (and the longest mail I have written for quite some time :D), if I still didn't convince anyone that this is a bad idea, I am willing to adjust my views according to the majority ones. I'm also trying the same, this way we can hope to arrive at the solution that is actually best

Best wishes,

Kevin

PS: I saw that you and Manuel went through another iteration of mails in the meantime but I feel what I wrote still has relevance

Best, Tom

Tom Bocklisch

mobile: +49 176 4206 1357 web: http://scm.iohttp://www.scalableminds.com/

scalable minds UG (haftungsbeschränkt) & Co. KG Stahnsdorfer Straße 152a | 14482 Potsdam

Handelsregister: HRA 5753 Registergericht Potsdam

Geschäftsführer: Tom Bocklisch, Tom Herold, Norman Rzepka, Thomas Werkmeister

Am 18. August 2016 um 17:51:33, Manuel Berning (notifications@github.commailto:notifications@github.com) schrieb:

I do not think it is an instruction problem. Some people for example just keep flying straight the whole way (maybe they do not care or sth?). But if you think you can get them all, please work on the movie and let us know when you found a version that works for everybody. Why not let them trace? -> Because somebody has to pay for it (the problem is not paying for this test but estimating what it would take in real use case).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/scalableminds/oxalis/issues/1440#issuecomment-240767683, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABDCrP4mhJ6c6XNTSbIwx_GzNpsFsedRks5qhH9-gaJpZM4Jlzl7.

[tmbo]

please keep Heiko on the email correspondences

Moritz Helmstaedter

Department of Connectomics Max Planck Institute for Brain Research Max-von-Laue-Str. 4 D-60438 Frankfurt, Germany

mh@brain.mpg.demailto:mh@brain.mpg.de www.brain.mpg.de/helmstaedterhttp://www.brain.mpg.de/helmstaedter

Department office: mhoffice@brain.mpg.demailto:mhoffice@brain.mpg.de (ph) +49 69 850033 3001

From: Moritz Helmstaedter Sent: Friday, August 19, 2016 12:27 PM To: Kevin Boergens Cc: Manuel Berning; scalableminds/oxalis; Mention; scalableminds/oxalis; Manuel Berning; Tom Bocklisch Subject: RE: Call at 2pm today Re: [scalableminds/oxalis] Connect RESCOP Service to validate annotation quality (#1440)

I can do 1:15p not sure about others?

Moritz Helmstaedter

Department of Connectomics Max Planck Institute for Brain Research Max-von-Laue-Str. 4 D-60438 Frankfurt, Germany

mh@brain.mpg.demailto:mh@brain.mpg.de www.brain.mpg.de/helmstaedterhttp://www.brain.mpg.de/helmstaedter

Department office: mhoffice@brain.mpg.demailto:mhoffice@brain.mpg.de (ph) +49 69 850033 3001

From: Kevin Boergens [mailto:kevin.boergens@brain.mpg.de] Sent: Friday, August 19, 2016 12:26 PM To: Moritz Helmstaedter Cc: Manuel Berning; scalableminds/oxalis; Mention; scalableminds/oxalis; Manuel Berning; Tom Bocklisch Subject: Re: Call at 2pm today Re: [scalableminds/oxalis] Connect RESCOP Service to validate annotation quality (#1440)

Hi,

sorry premature sending. I'm at a doctor's appointment later so 2pm is a bit tight.

Best, Kevin

On Aug 19, 2016 8:07 AM, "Moritz Helmstaedter" moritz.helmstaedter@brain.mpg.de<mailto:moritz.helmstaedter@brain.mpg.de> wrote: Hi guys

Let's discuss today at 2pm if possible please. (All aspects, including the one below)

Best,

m

Moritz Helmstaedter

Department of Connectomics Max Planck Institute for Brain Research Max-von-Laue Str. 4, 60316 Frankfurt, Germany www.brain.mpg.de/helmstaedterhttp://www.brain.mpg.de/helmstaedter Department office: mhoffice@brain.mpg.demailto:mhoffice@brain.mpg.de (ph) +49 69 850033 3001tel:+49%2069%20850033%203001

On 18 Aug 2016, at 22:12, Kevin Boergens kevin.boergens@brain.mpg.de<mailto:kevin.boergens@brain.mpg.de> wrote: Hi Tom,

thanks for your mail! Lively discussion is always appreciated :-)

On 18 August 2016 at 16:28, Tom Bocklisch tom.bocklisch@scalableminds.com<mailto:tom.bocklisch@scalableminds.com> wrote: Don't get me wrong, I don not want to keep everyone as a tracer. We got the great possibility to ban tracers from taking new assignments. But I just don't see the point of doing that in the middle of an assignment because: Actually I'm not super keen on banning. If you have a robust payment filter there is nothing wrong with encouraging to give it another go (they watch the video again and perhaps then the "coin drops")

• Way harder to implement (needs quite some front end changes, backend as well) I'm not convinced it would take backend changes. I think Manuel is a big boy and can live with node data as JSON instead of XML. I'm happy to do the frontend changes.
• Does not save money. We can still reject the work afterwards and we won't pay a cent more, there is no benefit in doing it in the middle of the assignment Actually there is. So the way it works is that tasks are layered with slightly overlapping bounding boxes. The "ground truth" of the new task is the consensus of the old. We need this filter, otherwise our consensus building doesn't work. So we know after 2um whether we will look at a result at all. And still we would let the person continue for another 8 minutes without the slightest chance of payment? That sounds cruel. Also, for new annotators it is shown what our payout fraction is. So if someone quits the tasks after two minutes, that is much better for us than if they submit and we don't pay them
• Checking the quality in the backend is simpler and more secure I thought about that. It is of course conceivable that someone would find a way to ignore the Result coming back from RESCOPaaS. But we would still save the result and then we could still block the payment for those cases (we would have to do that anyway in case they just wait it out and press submit after 10mins.

And remember, the crowd that is working at mturk needs to keep their so called reputation score up, because every good paying job looks at the percentage of faulty assignments the worker made. Hence, I think the percentage of people maliciously trying to cheat is low. That's exactly my point. If somebody wants to try again, they are welcome to and we should treat them with dignity and let them know as soon as we know that there is no chance of a payout.

Getting back to your last statement "the problem is not paying for this test but estimating what it would take in real use case": Keeping a side that the backend solution is actually cheaper. I, again, strongly disagree. Right know the priority is the paper, right? We are running on a tight schedule, so if there is no benefit in having this more involved implementation at the moment leet's keep focused on what is next.

Thats it from my side (and the longest mail I have written for quite some time :D), if I still didn't convince anyone that this is a bad idea, I am willing to adjust my views according to the majority ones. I'm also trying the same, this way we can hope to arrive at the solution that is actually best

Best wishes,

Kevin

PS: I saw that you and Manuel went through another iteration of mails in the meantime but I feel what I wrote still has relevance

Best, Tom

Tom Bocklisch

mobile: +49 176 4206 1357 web: http://scm.iohttp://www.scalableminds.com/

scalable minds UG (haftungsbeschränkt) & Co. KG Stahnsdorfer Straße 152a | 14482 Potsdam

Handelsregister: HRA 5753 Registergericht Potsdam

Geschäftsführer: Tom Bocklisch, Tom Herold, Norman Rzepka, Thomas Werkmeister

Am 18. August 2016 um 17:51:33, Manuel Berning (notifications@github.commailto:notifications@github.com) schrieb:

I do not think it is an instruction problem. Some people for example just keep flying straight the whole way (maybe they do not care or sth?). But if you think you can get them all, please work on the movie and let us know when you found a version that works for everybody. Why not let them trace? -> Because somebody has to pay for it (the problem is not paying for this test but estimating what it would take in real use case).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/scalableminds/oxalis/issues/1440#issuecomment-240767683, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABDCrP4mhJ6c6XNTSbIwx_GzNpsFsedRks5qhH9-gaJpZM4Jlzl7.

[tmbo]

intro movie ortho mode done, Heiko is uploading so we can have a look before the call.

pls activate flight tasks for me.

best M

Moritz Helmstaedter

Department of Connectomics Max Planck Institute for Brain Research Max-von-Laue-Str. 4 D-60438 Frankfurt, Germany

mh@brain.mpg.demailto:mh@brain.mpg.de www.brain.mpg.de/helmstaedterhttp://www.brain.mpg.de/helmstaedter

Department office: mhoffice@brain.mpg.demailto:mhoffice@brain.mpg.de (ph) +49 69 850033 3001

From: Webknossos [mailto:webknossos-bounces@brain.mpg.de] On Behalf Of Moritz Helmstaedter via Webknossos Sent: Friday, August 19, 2016 12:57 PM To: Kevin Boergens Cc: scalableminds/oxalis; Mention; Tom Bocklisch; scalableminds/oxalis; webknossos@brain.mpg.de; Manuel Berning Subject: Re: [Webknossos] Call at 2pm today Re: [scalableminds/oxalis] Connect RESCOP Service to validate annotation quality (#1440)

please keep Heiko on the email correspondences

Moritz Helmstaedter

Department of Connectomics Max Planck Institute for Brain Research Max-von-Laue-Str. 4 D-60438 Frankfurt, Germany

mh@brain.mpg.demailto:mh@brain.mpg.de www.brain.mpg.de/helmstaedterhttp://www.brain.mpg.de/helmstaedter

Department office: mhoffice@brain.mpg.demailto:mhoffice@brain.mpg.de (ph) +49 69 850033 3001

From: Moritz Helmstaedter Sent: Friday, August 19, 2016 12:27 PM To: Kevin Boergens Cc: Manuel Berning; scalableminds/oxalis; Mention; scalableminds/oxalis; Manuel Berning; Tom Bocklisch Subject: RE: Call at 2pm today Re: [scalableminds/oxalis] Connect RESCOP Service to validate annotation quality (#1440)

I can do 1:15p not sure about others?

Moritz Helmstaedter

Department of Connectomics Max Planck Institute for Brain Research Max-von-Laue-Str. 4 D-60438 Frankfurt, Germany

mh@brain.mpg.demailto:mh@brain.mpg.de www.brain.mpg.de/helmstaedterhttp://www.brain.mpg.de/helmstaedter

Department office: mhoffice@brain.mpg.demailto:mhoffice@brain.mpg.de (ph) +49 69 850033 3001

From: Kevin Boergens [mailto:kevin.boergens@brain.mpg.de] Sent: Friday, August 19, 2016 12:26 PM To: Moritz Helmstaedter Cc: Manuel Berning; scalableminds/oxalis; Mention; scalableminds/oxalis; Manuel Berning; Tom Bocklisch Subject: Re: Call at 2pm today Re: [scalableminds/oxalis] Connect RESCOP Service to validate annotation quality (#1440)

Hi,

sorry premature sending. I'm at a doctor's appointment later so 2pm is a bit tight.

Best, Kevin

On Aug 19, 2016 8:07 AM, "Moritz Helmstaedter" moritz.helmstaedter@brain.mpg.de<mailto:moritz.helmstaedter@brain.mpg.de> wrote: Hi guys

Let's discuss today at 2pm if possible please. (All aspects, including the one below)

Best,

m

Moritz Helmstaedter

Department of Connectomics Max Planck Institute for Brain Research Max-von-Laue Str. 4, 60316 Frankfurt, Germany www.brain.mpg.de/helmstaedterhttp://www.brain.mpg.de/helmstaedter Department office: mhoffice@brain.mpg.demailto:mhoffice@brain.mpg.de (ph) +49 69 850033 3001tel:+49%2069%20850033%203001

On 18 Aug 2016, at 22:12, Kevin Boergens kevin.boergens@brain.mpg.de<mailto:kevin.boergens@brain.mpg.de> wrote: Hi Tom,

thanks for your mail! Lively discussion is always appreciated :-)

On 18 August 2016 at 16:28, Tom Bocklisch tom.bocklisch@scalableminds.com<mailto:tom.bocklisch@scalableminds.com> wrote: Don't get me wrong, I don not want to keep everyone as a tracer. We got the great possibility to ban tracers from taking new assignments. But I just don't see the point of doing that in the middle of an assignment because: Actually I'm not super keen on banning. If you have a robust payment filter there is nothing wrong with encouraging to give it another go (they watch the video again and perhaps then the "coin drops")

• Way harder to implement (needs quite some front end changes, backend as well) I'm not convinced it would take backend changes. I think Manuel is a big boy and can live with node data as JSON instead of XML. I'm happy to do the frontend changes.
• Does not save money. We can still reject the work afterwards and we won't pay a cent more, there is no benefit in doing it in the middle of the assignment Actually there is. So the way it works is that tasks are layered with slightly overlapping bounding boxes. The "ground truth" of the new task is the consensus of the old. We need this filter, otherwise our consensus building doesn't work. So we know after 2um whether we will look at a result at all. And still we would let the person continue for another 8 minutes without the slightest chance of payment? That sounds cruel. Also, for new annotators it is shown what our payout fraction is. So if someone quits the tasks after two minutes, that is much better for us than if they submit and we don't pay them
• Checking the quality in the backend is simpler and more secure I thought about that. It is of course conceivable that someone would find a way to ignore the Result coming back from RESCOPaaS. But we would still save the result and then we could still block the payment for those cases (we would have to do that anyway in case they just wait it out and press submit after 10mins.

And remember, the crowd that is working at mturk needs to keep their so called reputation score up, because every good paying job looks at the percentage of faulty assignments the worker made. Hence, I think the percentage of people maliciously trying to cheat is low. That's exactly my point. If somebody wants to try again, they are welcome to and we should treat them with dignity and let them know as soon as we know that there is no chance of a payout.

Getting back to your last statement "the problem is not paying for this test but estimating what it would take in real use case": Keeping a side that the backend solution is actually cheaper. I, again, strongly disagree. Right know the priority is the paper, right? We are running on a tight schedule, so if there is no benefit in having this more involved implementation at the moment leet's keep focused on what is next.

Thats it from my side (and the longest mail I have written for quite some time :D), if I still didn't convince anyone that this is a bad idea, I am willing to adjust my views according to the majority ones. I'm also trying the same, this way we can hope to arrive at the solution that is actually best

Best wishes,

Kevin

PS: I saw that you and Manuel went through another iteration of mails in the meantime but I feel what I wrote still has relevance

Best, Tom

Tom Bocklisch

mobile: +49 176 4206 1357 web: http://scm.iohttp://www.scalableminds.com/

scalable minds UG (haftungsbeschränkt) & Co. KG Stahnsdorfer Straße 152a | 14482 Potsdam

Handelsregister: HRA 5753 Registergericht Potsdam

Geschäftsführer: Tom Bocklisch, Tom Herold, Norman Rzepka, Thomas Werkmeister

Am 18. August 2016 um 17:51:33, Manuel Berning (notifications@github.commailto:notifications@github.com) schrieb:

I do not think it is an instruction problem. Some people for example just keep flying straight the whole way (maybe they do not care or sth?). But if you think you can get them all, please work on the movie and let us know when you found a version that works for everybody. Why not let them trace? -> Because somebody has to pay for it (the problem is not paying for this test but estimating what it would take in real use case).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/scalableminds/oxalis/issues/1440#issuecomment-240767683, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABDCrP4mhJ6c6XNTSbIwx_GzNpsFsedRks5qhH9-gaJpZM4Jlzl7.

[hotzenklotz]

This is no longer relevant right now. Please reopen when needed.