ipeirotis / Troia-Server

Quality Control API for Crowdsourcing Applications
http://project-troia.com/
Apache License 2.0
15 stars 9 forks source link

Worker Quality is NaN #88

Open PeccatorImpius opened 11 years ago

PeccatorImpius commented 11 years ago

For a number of our jobs, I am getting worker quality as NaN for all workers. Please see comments below for steps to reproduce.

PeccatorImpius commented 11 years ago

Step 1:

$ curl -X POST -H "Content-Type: application/json" "http://localhost/troia-server-1.1/jobs" -d '{"categories":["blank","confirm_closed","confirm_no","confirm_nonrestaurant","confirm_yes"],"costMatrix":[{"from":"blank","to":"blank","value":5.0},{"from":"blank","to":"confirm_closed","value":5.0},{"from":"blank","to":"confirm_no","value":5.0},{"from":"blank","to":"confirm_nonrestaurant","value":5.0},{"from":"blank","to":"confirm_yes","value":5.0},{"from":"confirm_closed","to":"blank","value":5.0},{"from":"confirm_closed","to":"confirm_closed","value":0.0},{"from":"confirm_closed","to":"confirm_no","value":5.0},{"from":"confirm_closed","to":"confirm_nonrestaurant","value":5.0},{"from":"confirm_closed","to":"confirm_yes","value":5.0},{"from":"confirm_no","to":"blank","value":1.0},{"from":"confirm_no","to":"confirm_closed","value":1.0},{"from":"confirm_no","to":"confirm_no","value":0.0},{"from":"confirm_no","to":"confirm_nonrestaurant","value":1.0},{"from":"confirm_no","to":"confirm_yes","value":1.0},{"from":"confirm_nonrestaurant","to":"blank","value":5.0},{"from":"confirm_nonrestaurant","to":"confirm_closed","value":5.0},{"from":"confirm_nonrestaurant","to":"confirm_no","value":5.0},{"from":"confirm_nonrestaurant","to":"confirm_nonrestaurant","value":0.0},{"from":"confirm_nonrestaurant","to":"confirm_yes","value":5.0},{"from":"confirm_yes","to":"blank","value":1.0},{"from":"confirm_yes","to":"confirm_closed","value":1.0},{"from":"confirm_yes","to":"confirm_no","value":1.0},{"from":"confirm_yes","to":"confirm_nonrestaurant","value":1.0},{"from":"confirm_yes","to":"confirm_yes","value":0.0}],"algorithm":"IDS","iterations":10,"epsilon":0.0001,"scheduler":"NormalScheduler","calculator":"CostBased"}' {"timestamp":"2013-05-11T16:02:06.659-07:00","result":"New job created with ID: RANDOM__1368313326658_39","status":"OK"}

PeccatorImpius commented 11 years ago

Step 2:

$ curl -X POST -H "Content-Type: application/json" "http://localhost/troia-server-1.1/jobs/RANDOM__1368313326658_39/assigns" -d '{"assigns":[{"label":"confirm_yes","worker":"A1R7CJMWXC79UO","object":"2OVZKPFE4EGD044XHZZIHNZWGWV7HC"},{"label":"confirm_yes","worker":"A2VRQML8Q3XYP4","object":"2OVZKPFE4EGD044XHZZIHNZWGWV7HC"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2OVZKPFE4EGD044XHZZIHNZWGWV7HC"},{"label":"confirm_yes","worker":"A3TXO6RKFIDFUV","object":"2Z3KH1Q6SVQ80ZQH28XLDOBVMXHL2G"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2Z3KH1Q6SVQ80ZQH28XLDOBVMXHL2G"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2Z3KH1Q6SVQ80ZQH28XLDOBVMXHL2G"},{"label":"confirm_yes","worker":"A1FQYUBCBNTQX7","object":"2V6ZFYQS1CST5FXS3RJ4QC1E8S3KN3"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2V6ZFYQS1CST5FXS3RJ4QC1E8S3KN3"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2V6ZFYQS1CST5FXS3RJ4QC1E8S3KN3"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2MCFJ51Y7QEOI6GL4F3S1MOF76TBEL"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2MCFJ51Y7QEOI6GL4F3S1MOF76TBEL"},{"label":"confirm_yes","worker":"A3TXO6RKFIDFUV","object":"2MCFJ51Y7QEOI6GL4F3S1MOF76TBEL"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2S4JTUHYW2GT8095J6WWU169874PK4"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2S4JTUHYW2GT8095J6WWU169874PK4"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2S4JTUHYW2GT8095J6WWU169874PK4"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2S3YHVI44OS23CC3P15Z0BAPFQI4YO"},{"label":"confirm_closed","worker":"A31X3JCHS0BPFJ","object":"2S3YHVI44OS23CC3P15Z0BAPFQI4YO"},{"label":"confirm_yes","worker":"AH5ZIMHL7TNWC","object":"2S3YHVI44OS23CC3P15Z0BAPFQI4YO"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2KXDEY5COW0LZIHYKQIFTC6VX9G4VF"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2KXDEY5COW0LZIHYKQIFTC6VX9G4VF"},{"label":"confirm_yes","worker":"A2KET1HL1COET5","object":"2KXDEY5COW0LZIHYKQIFTC6VX9G4VF"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2FT49F9SSSHXKS1JZ6K5P5F8TJ4TX2"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2FT49F9SSSHXKS1JZ6K5P5F8TJ4TX2"},{"label":"confirm_no","worker":"A3TXO6RKFIDFUV","object":"2FT49F9SSSHXKS1JZ6K5P5F8TJ4TX2"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"27G2RCJK63ZVXUZMCYLIIQ9JVK87WC"},{"label":"confirm_yes","worker":"A19BL9GZGXFWFT","object":"27G2RCJK63ZVXUZMCYLIIQ9JVK87WC"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"27G2RCJK63ZVXUZMCYLIIQ9JVK87WC"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2JODG67D1X3VJ62VO2B2RE1MH40HAC"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2JODG67D1X3VJ62VO2B2RE1MH40HAC"},{"label":"confirm_yes","worker":"A2VRQML8Q3XYP4","object":"2JODG67D1X3VJ62VO2B2RE1MH40HAC"},{"label":"confirm_closed","worker":"A31X3JCHS0BPFJ","object":"26ZIT3HVWAVK1XKIV4T1F2Z9J5XEMD"},{"label":"confirm_closed","worker":"A3TXO6RKFIDFUV","object":"26ZIT3HVWAVK1XKIV4T1F2Z9J5XEMD"},{"label":"confirm_nonrestaurant","worker":"AH5ZIMHL7TNWC","object":"26ZIT3HVWAVK1XKIV4T1F2Z9J5XEMD"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"20N1Y7QEOZFY9JJ747DONXRD2JYEH4"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"20N1Y7QEOZFY9JJ747DONXRD2JYEH4"},{"label":"confirm_yes","worker":"A2KET1HL1COET5","object":"20N1Y7QEOZFY9JJ747DONXRD2JYEH4"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"241KM05BMJTU0PEXS7G9ZA7SFB2EJ9"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"241KM05BMJTU0PEXS7G9ZA7SFB2EJ9"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"241KM05BMJTU0PEXS7G9ZA7SFB2EJ9"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2YDWAVKI62NJ9TJ2ED09YH6BU0SRJZ"},{"label":"confirm_closed","worker":"A19BL9GZGXFWFT","object":"2YDWAVKI62NJ9TJ2ED09YH6BU0SRJZ"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2YDWAVKI62NJ9TJ2ED09YH6BU0SRJZ"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2JV21O3W5XH02G7NUGBYMPLSZMLHB7"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2JV21O3W5XH02G7NUGBYMPLSZMLHB7"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2JV21O3W5XH02G7NUGBYMPLSZMLHB7"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2JS1QP6AUC26W7O2PFO330FKAR5811"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2JS1QP6AUC26W7O2PFO330FKAR5811"},{"label":"confirm_yes","worker":"A1I3CXC17NIRWB","object":"2JS1QP6AUC26W7O2PFO330FKAR5811"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2CIFK0COK2JEKDPKWY0LZW6O9PSRKY"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2CIFK0COK2JEKDPKWY0LZW6O9PSRKY"},{"label":"confirm_yes","worker":"A2KET1HL1COET5","object":"2CIFK0COK2JEKDPKWY0LZW6O9PSRKY"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2IJK274J79KQFXJ3ZIXU5FCDEKDHFP"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2IJK274J79KQFXJ3ZIXU5FCDEKDHFP"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2IJK274J79KQFXJ3ZIXU5FCDEKDHFP"},{"label":"confirm_closed","worker":"A3TXO6RKFIDFUV","object":"234EGOOGQSCMP9S5E65I2UU0A36EEO"},{"label":"confirm_closed","worker":"A3PRF4IIM2IQN2","object":"234EGOOGQSCMP9S5E65I2UU0A36EEO"},{"label":"confirm_closed","worker":"A31X3JCHS0BPFJ","object":"234EGOOGQSCMP9S5E65I2UU0A36EEO"},{"label":"confirm_yes","worker":"A3TXO6RKFIDFUV","object":"23QJIA0RYNJ9LE1FYEWBUJTURDD27P"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"23QJIA0RYNJ9LE1FYEWBUJTURDD27P"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"23QJIA0RYNJ9LE1FYEWBUJTURDD27P"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2S5VP3TVOK5UYANWALHEWZFY07I52D"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2S5VP3TVOK5UYANWALHEWZFY07I52D"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2S5VP3TVOK5UYANWALHEWZFY07I52D"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2ETAT2D5NQYNO6LLCX751YMNMBF84E"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2ETAT2D5NQYNO6LLCX751YMNMBF84E"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2ETAT2D5NQYNO6LLCX751YMNMBF84E"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2Z2MN9U8P9Y3RKER9WUS2YIMAY9EVU"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2Z2MN9U8P9Y3RKER9WUS2YIMAY9EVU"},{"label":"confirm_yes","worker":"A1I3CXC17NIRWB","object":"2Z2MN9U8P9Y3RKER9WUS2YIMAY9EVU"},{"label":"confirm_yes","worker":"ABT7QTMIYXYO0","object":"2GXYQS1CSTMOYO984I9C9EYDXCZPM9"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2GXYQS1CSTMOYO984I9C9EYDXCZPM9"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2GXYQS1CSTMOYO984I9C9EYDXCZPM9"},{"label":"blank","worker":"AH5ZIMHL7TNWC","object":"2BLK4F0OHOVRKV0SW2TLH2HEMBR23K"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2BLK4F0OHOVRKV0SW2TLH2HEMBR23K"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2BLK4F0OHOVRKV0SW2TLH2HEMBR23K"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2X6OGQSCM6IATTA9U8LU80OQUZQHHN"},{"label":"confirm_yes","worker":"A2KET1HL1COET5","object":"2X6OGQSCM6IATTA9U8LU80OQUZQHHN"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2X6OGQSCM6IATTA9U8LU80OQUZQHHN"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"29UWY2AOO2GM55YJ0UHSKM6IKPI442"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"29UWY2AOO2GM55YJ0UHSKM6IKPI442"},{"label":"confirm_yes","worker":"AH5ZIMHL7TNWC","object":"29UWY2AOO2GM55YJ0UHSKM6IKPI442"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2DJVP9746OQ1BE8LJ4X7851QUR3L1Z"},{"label":"confirm_yes","worker":"ABT7QTMIYXYO0","object":"2DJVP9746OQ1BE8LJ4X7851QUR3L1Z"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2DJVP9746OQ1BE8LJ4X7851QUR3L1Z"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"25XXRDS4IC1EH45SVTD19A2I5DKWZS"},{"label":"confirm_no","worker":"A3PRF4IIM2IQN2","object":"25XXRDS4IC1EH45SVTD19A2I5DKWZS"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"25XXRDS4IC1EH45SVTD19A2I5DKWZS"},{"label":"confirm_yes","worker":"A2VRQML8Q3XYP4","object":"2DABRI8IUB2YD0QET6KLJ3L0PQZH56"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2DABRI8IUB2YD0QET6KLJ3L0PQZH56"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2DABRI8IUB2YD0QET6KLJ3L0PQZH56"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2TOSK9RJ85OZ1QWCYO1PUOOQFTCU60"},{"label":"confirm_yes","worker":"AH5ZIMHL7TNWC","object":"2TOSK9RJ85OZ1QWCYO1PUOOQFTCU60"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2TOSK9RJ85OZ1QWCYO1PUOOQFTCU60"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2IBHV8ER9Y8T6B0HB6D5SFHSC334RM"},{"label":"confirm_yes","worker":"ABT7QTMIYXYO0","object":"2IBHV8ER9Y8T6B0HB6D5SFHSC334RM"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2IBHV8ER9Y8T6B0HB6D5SFHSC334RM"},{"label":"confirm_yes","worker":"A3TXO6RKFIDFUV","object":"294EZZ2MIKMNSLQKLCU81WWXSI97O0"},{"label":"confirm_yes","worker":"A19BL9GZGXFWFT","object":"294EZZ2MIKMNSLQKLCU81WWXSI97O0"},{"label":"confirm_yes","worker":"A1I3CXC17NIRWB","object":"294EZZ2MIKMNSLQKLCU81WWXSI97O0"},{"label":"confirm_yes","worker":"A2VRQML8Q3XYP4","object":"2AOYTWX4H3H282M8LN7IEIJRLKM4ZF"}]}' {"timestamp":"2013-05-11T16:03:34.374-07:00","status":"OK","redirect":"responses/3183/POST/jobs/RANDOM__1368313326658_39/assigns"}

PeccatorImpius commented 11 years ago

Step 3:

$ curl -X GET "http://localhost/troia-server-1.1/responses/3183/POST/jobs/RANDOM__1368313326658_39/assigns" {"timestamp":"2013-05-11T16:04:22.189-07:00","result":"Assigns added","executionTime":0.002,"status":"OK"}

PeccatorImpius commented 11 years ago

Step 4:

$ curl -X GET "http://localhost/troia-server-1.1/jobs/RANDOM__1368313326658_39/workers/quality/estimated" {"timestamp":"2013-05-11T16:05:21.667-07:00","status":"OK","redirect":"responses/3204/GET/jobs/RANDOM__1368313326658_39/workers/quality/estimated"}

PeccatorImpius commented 11 years ago

Step 5:

$ curl -X GET "http://localhost/troia-server-1.1/responses/3204/GET/jobs/RANDOM__1368313326658_39/workers/quality/estimated" {"timestamp":"2013-05-11T16:06:01.420-07:00","result":[{"workerName":"A2KET1HL1COET5","value":NaN},{"workerName":"A31X3JCHS0BPFJ","value":NaN},{"workerName":"A3PRF4IIM2IQN2","value":NaN},{"workerName":"A1FQYUBCBNTQX7","value":NaN},{"workerName":"ABT7QTMIYXYO0","value":NaN},{"workerName":"A2L2ZXRHK3SNOP","value":NaN},{"workerName":"A2VRQML8Q3XYP4","value":NaN},{"workerName":"A1R7CJMWXC79UO","value":NaN},{"workerName":"A19BL9GZGXFWFT","value":NaN},{"workerName":"AH5ZIMHL7TNWC","value":NaN},{"workerName":"A1I3CXC17NIRWB","value":NaN},{"workerName":"A3TXO6RKFIDFUV","value":NaN}],"executionTime":0.0,"status":"OK"}

kkonrad commented 11 years ago

Thanks for reporting this issue! Dana have turned it into test. We will let you know when we manage to fix this.

kkonrad commented 11 years ago

Quick update:

It looks like there is some problem with dynamic prior calculations done by IDS algorithm. We are still working on this.

PeccatorImpius commented 11 years ago

Thanks for the update. The workaround is very helpful. I will try to use priors while I am waiting for the fix.

PeccatorImpius commented 11 years ago

I have implemented the workaround with priors, but I am seeing some strange numbers:

$ curl -X GET "http://localhost/troia-server-1.1/responses/1148/GET/jobs/RANDOM__1368726379654_33/workers/quality/estimated"
{"timestamp":"2013-05-16T10:53:46.686-07:00","result":[{"workerName":"A1W0N2QACQZEQP","value":-3.1727002269058184},{"workerName":"A224TK7J4KA1LV","value":-3.3285464160473195},{"workerName":"A1GU6OYGOV6PBK","value":0.12620888888888937},{"workerName":"AHHKIENAC9M3M","value":0.6288671324385611},{"workerName":"A239WM193Q1OIY","value":-3.3466950270926326}],"executionTime":0.0,"status":"OK"}

Is it normal for worker quality numbers to be negative?

kkonrad commented 11 years ago

It works that way: quality is computed as 1 - (worker_cost / spammer_cost). workers cost is expected costs that he will generate making an assign. Our baseline is spammer_cost which is using priors and costs to select label which minimizes cost. So in this case spammer is generating lower costs than workers ... So it looks like quality can be negative. Nevertheless, I will ask someone smarter than me about this ;) BTW As you can see changing priors will result in different worker qualities. More accurate priors should result in better worker quality estimation

kkonrad commented 11 years ago

It is confirmed that in some cases worker quality can be negative. It looks like that this is not enough data for IDS to coverage and it is recommended for such small jobs use BDS algorithm. IDS works well with high volume of assigns. We won't tackle this problem any time soon.