StanfordHCI / fairwork

Server for the Fair Work Mechanical Turk script
MIT License
7 stars 3 forks source link

Should we change freeze to ignore? #33

Open markwhiting opened 4 years ago

markwhiting commented 4 years ago

Freeze doesn't capture what's happening, because actually we're just not paying attention to the worker's reports, we're not literally freezing them, and it doesn't freeze bonuses either.

Ignore better captures it.

mbernst commented 4 years ago

"You can review the pending bonuses below and freeze bonuses if something looks unusual. Please remember to trust the workers' estimates, and only freeze bonuses if absolutely needed."

This feels similar to me to the notion of freezing payment. I'm less sure how we'd word this if it were about ignoring — ignore the worker's report?

markwhiting commented 4 years ago

Something like:

You can review the pending bonuses below and click to ignore reports that appear unusual. Please remember to trust the workers' estimates, and only ignore reports when its clear that something is not correct.

mbernst commented 4 years ago

"Freeze" also makes clear that the worker doesn't get paid the bonus. I wonder how much we need to be clear about that here too.

On Sat, Mar 21, 2020 at 6:33 PM Mark Whiting notifications@github.com wrote:

Something like:

You can review the pending bonuses below and click to ignore reports that appear unusual. Please remember to trust the workers' estimates, and only ignore reports when its clear that something is not correct.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/StanfordHCI/fairwork/issues/33#issuecomment-602133075, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB43ROMCCJYJMYF7YWPJ6DRIVTFTANCNFSM4LQTYMYQ .

markwhiting commented 4 years ago

Ah, Grant and I were discussing this and thought the worker does get the bonus, they just don’t get their time considered.

On Mar 21, 2020, at 21:42, Michael Bernstein notifications@github.com wrote:

"Freeze" also makes clear that the worker doesn't get paid the bonus. I wonder how much we need to be clear about that here too.

On Sat, Mar 21, 2020 at 6:33 PM Mark Whiting notifications@github.com wrote:

Something like:

You can review the pending bonuses below and click to ignore reports that appear unusual. Please remember to trust the workers' estimates, and only ignore reports when its clear that something is not correct.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/StanfordHCI/fairwork/issues/33#issuecomment-602133075, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB43ROMCCJYJMYF7YWPJ6DRIVTFTANCNFSM4LQTYMYQ .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/StanfordHCI/fairwork/issues/33#issuecomment-602134092, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFELE27AEEMEQQQH64WV63RIVUJZANCNFSM4LQTYMYQ.

mbernst commented 4 years ago

Oh! I forget.

On Sat, Mar 21, 2020 at 6:44 PM Mark Whiting notifications@github.com wrote:

Ah, Grant and I were discussing this and thought the worker does get the bonus, they just don’t get their time considered.

On Mar 21, 2020, at 21:42, Michael Bernstein notifications@github.com wrote:

"Freeze" also makes clear that the worker doesn't get paid the bonus. I wonder how much we need to be clear about that here too.

On Sat, Mar 21, 2020 at 6:33 PM Mark Whiting notifications@github.com wrote:

Something like:

You can review the pending bonuses below and click to ignore reports that appear unusual. Please remember to trust the workers' estimates, and only ignore reports when its clear that something is not correct.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/StanfordHCI/fairwork/issues/33#issuecomment-602133075>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAB43ROMCCJYJMYF7YWPJ6DRIVTFTANCNFSM4LQTYMYQ

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/StanfordHCI/fairwork/issues/33#issuecomment-602134092>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAFELE27AEEMEQQQH64WV63RIVUJZANCNFSM4LQTYMYQ .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/StanfordHCI/fairwork/issues/33#issuecomment-602134256, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB43RJU2YDKKPO3YWLW3X3RIVUQNANCNFSM4LQTYMYQ .

-- sent from my NLS

markwhiting commented 4 years ago

Just checked, and I think it is paying frozen workers. Here's where the list of workers to be paid is constructed:

https://github.com/StanfordHCI/fairwork/blob/735db0360dc29e26a67f28a6ee59831abe1172e2/auditor/management/commands/payaudits.py#L54

tnarg999 commented 4 years ago

If my memory is right, we had a meeting last year where we decided that it was best to still pay frozen workers because it would essentially pay the worker the amount they would have gotten if they had reported a more reasonable time. I think we also noted that it discourages requesters from freezing simply because they don't want to pay bonuses.

Do we want to continue paying frozen workers or do we want to reconsider this question?

mbernst commented 4 years ago

Yes, absolutely!

On Wed, Mar 25, 2020 at 7:27 PM Grant Hugh notifications@github.com wrote:

If my memory is right, we had a meeting last year where we decided that it was best to still pay frozen workers because it would essentially pay the worker the amount they would have gotten if they had reported a more reasonable time. I think we also noted that it discourages requesters from freezing simply because they don't want to pay bonuses.

Do we want to continue paying frozen workers or do we want to reconsider this question?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/StanfordHCI/fairwork/issues/33#issuecomment-604191701, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB43RNS75PQRKP25ZPACKTRJK4QZANCNFSM4LQTYMYQ .

-- sent from my NLS

markwhiting commented 4 years ago

To me, paying frozen workers is still the correct behavior.

But, if we are paying them, I think it reinforces the idea of changing the term to something less cold.

tnarg999 commented 4 years ago

For sure, right now the language in the website and the emails definitely makes it sound like the worker won't get paid a bonus when they are frozen.

mbernst commented 4 years ago

Maybe "mute this worker's report" or "remove this worker's report"? "Ignore" feels a bit dismissive and also doesn't make clear what happens.

markwhiting commented 4 years ago

If I remember correctly its actually all reports from that worker for that requester, right?

So perhaps:

Omit worker's reports

I think its worth being as clear as possible here, and perhaps even saying a little more. e.g.,

Potential error in duration report. Omit reports from this worker.