Open PeccatorImpius opened 11 years ago
Step 1:
$ curl -X POST -H "Content-Type: application/json" "http://localhost/troia-server-1.1/jobs" -d '{"categories":["blank","confirm_closed","confirm_no","confirm_nonrestaurant","confirm_yes"],"costMatrix":[{"from":"blank","to":"blank","value":5.0},{"from":"blank","to":"confirm_closed","value":5.0},{"from":"blank","to":"confirm_no","value":5.0},{"from":"blank","to":"confirm_nonrestaurant","value":5.0},{"from":"blank","to":"confirm_yes","value":5.0},{"from":"confirm_closed","to":"blank","value":5.0},{"from":"confirm_closed","to":"confirm_closed","value":0.0},{"from":"confirm_closed","to":"confirm_no","value":5.0},{"from":"confirm_closed","to":"confirm_nonrestaurant","value":5.0},{"from":"confirm_closed","to":"confirm_yes","value":5.0},{"from":"confirm_no","to":"blank","value":1.0},{"from":"confirm_no","to":"confirm_closed","value":1.0},{"from":"confirm_no","to":"confirm_no","value":0.0},{"from":"confirm_no","to":"confirm_nonrestaurant","value":1.0},{"from":"confirm_no","to":"confirm_yes","value":1.0},{"from":"confirm_nonrestaurant","to":"blank","value":5.0},{"from":"confirm_nonrestaurant","to":"confirm_closed","value":5.0},{"from":"confirm_nonrestaurant","to":"confirm_no","value":5.0},{"from":"confirm_nonrestaurant","to":"confirm_nonrestaurant","value":0.0},{"from":"confirm_nonrestaurant","to":"confirm_yes","value":5.0},{"from":"confirm_yes","to":"blank","value":1.0},{"from":"confirm_yes","to":"confirm_closed","value":1.0},{"from":"confirm_yes","to":"confirm_no","value":1.0},{"from":"confirm_yes","to":"confirm_nonrestaurant","value":1.0},{"from":"confirm_yes","to":"confirm_yes","value":0.0}],"algorithm":"IDS","iterations":10,"epsilon":0.0001,"scheduler":"NormalScheduler","calculator":"CostBased"}' {"timestamp":"2013-05-11T16:02:06.659-07:00","result":"New job created with ID: RANDOM__1368313326658_39","status":"OK"}
Step 2:
$ curl -X POST -H "Content-Type: application/json" "http://localhost/troia-server-1.1/jobs/RANDOM__1368313326658_39/assigns" -d '{"assigns":[{"label":"confirm_yes","worker":"A1R7CJMWXC79UO","object":"2OVZKPFE4EGD044XHZZIHNZWGWV7HC"},{"label":"confirm_yes","worker":"A2VRQML8Q3XYP4","object":"2OVZKPFE4EGD044XHZZIHNZWGWV7HC"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2OVZKPFE4EGD044XHZZIHNZWGWV7HC"},{"label":"confirm_yes","worker":"A3TXO6RKFIDFUV","object":"2Z3KH1Q6SVQ80ZQH28XLDOBVMXHL2G"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2Z3KH1Q6SVQ80ZQH28XLDOBVMXHL2G"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2Z3KH1Q6SVQ80ZQH28XLDOBVMXHL2G"},{"label":"confirm_yes","worker":"A1FQYUBCBNTQX7","object":"2V6ZFYQS1CST5FXS3RJ4QC1E8S3KN3"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2V6ZFYQS1CST5FXS3RJ4QC1E8S3KN3"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2V6ZFYQS1CST5FXS3RJ4QC1E8S3KN3"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2MCFJ51Y7QEOI6GL4F3S1MOF76TBEL"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2MCFJ51Y7QEOI6GL4F3S1MOF76TBEL"},{"label":"confirm_yes","worker":"A3TXO6RKFIDFUV","object":"2MCFJ51Y7QEOI6GL4F3S1MOF76TBEL"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2S4JTUHYW2GT8095J6WWU169874PK4"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2S4JTUHYW2GT8095J6WWU169874PK4"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2S4JTUHYW2GT8095J6WWU169874PK4"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2S3YHVI44OS23CC3P15Z0BAPFQI4YO"},{"label":"confirm_closed","worker":"A31X3JCHS0BPFJ","object":"2S3YHVI44OS23CC3P15Z0BAPFQI4YO"},{"label":"confirm_yes","worker":"AH5ZIMHL7TNWC","object":"2S3YHVI44OS23CC3P15Z0BAPFQI4YO"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2KXDEY5COW0LZIHYKQIFTC6VX9G4VF"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2KXDEY5COW0LZIHYKQIFTC6VX9G4VF"},{"label":"confirm_yes","worker":"A2KET1HL1COET5","object":"2KXDEY5COW0LZIHYKQIFTC6VX9G4VF"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2FT49F9SSSHXKS1JZ6K5P5F8TJ4TX2"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2FT49F9SSSHXKS1JZ6K5P5F8TJ4TX2"},{"label":"confirm_no","worker":"A3TXO6RKFIDFUV","object":"2FT49F9SSSHXKS1JZ6K5P5F8TJ4TX2"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"27G2RCJK63ZVXUZMCYLIIQ9JVK87WC"},{"label":"confirm_yes","worker":"A19BL9GZGXFWFT","object":"27G2RCJK63ZVXUZMCYLIIQ9JVK87WC"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"27G2RCJK63ZVXUZMCYLIIQ9JVK87WC"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2JODG67D1X3VJ62VO2B2RE1MH40HAC"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2JODG67D1X3VJ62VO2B2RE1MH40HAC"},{"label":"confirm_yes","worker":"A2VRQML8Q3XYP4","object":"2JODG67D1X3VJ62VO2B2RE1MH40HAC"},{"label":"confirm_closed","worker":"A31X3JCHS0BPFJ","object":"26ZIT3HVWAVK1XKIV4T1F2Z9J5XEMD"},{"label":"confirm_closed","worker":"A3TXO6RKFIDFUV","object":"26ZIT3HVWAVK1XKIV4T1F2Z9J5XEMD"},{"label":"confirm_nonrestaurant","worker":"AH5ZIMHL7TNWC","object":"26ZIT3HVWAVK1XKIV4T1F2Z9J5XEMD"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"20N1Y7QEOZFY9JJ747DONXRD2JYEH4"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"20N1Y7QEOZFY9JJ747DONXRD2JYEH4"},{"label":"confirm_yes","worker":"A2KET1HL1COET5","object":"20N1Y7QEOZFY9JJ747DONXRD2JYEH4"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"241KM05BMJTU0PEXS7G9ZA7SFB2EJ9"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"241KM05BMJTU0PEXS7G9ZA7SFB2EJ9"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"241KM05BMJTU0PEXS7G9ZA7SFB2EJ9"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2YDWAVKI62NJ9TJ2ED09YH6BU0SRJZ"},{"label":"confirm_closed","worker":"A19BL9GZGXFWFT","object":"2YDWAVKI62NJ9TJ2ED09YH6BU0SRJZ"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2YDWAVKI62NJ9TJ2ED09YH6BU0SRJZ"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2JV21O3W5XH02G7NUGBYMPLSZMLHB7"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2JV21O3W5XH02G7NUGBYMPLSZMLHB7"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2JV21O3W5XH02G7NUGBYMPLSZMLHB7"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2JS1QP6AUC26W7O2PFO330FKAR5811"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2JS1QP6AUC26W7O2PFO330FKAR5811"},{"label":"confirm_yes","worker":"A1I3CXC17NIRWB","object":"2JS1QP6AUC26W7O2PFO330FKAR5811"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2CIFK0COK2JEKDPKWY0LZW6O9PSRKY"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2CIFK0COK2JEKDPKWY0LZW6O9PSRKY"},{"label":"confirm_yes","worker":"A2KET1HL1COET5","object":"2CIFK0COK2JEKDPKWY0LZW6O9PSRKY"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2IJK274J79KQFXJ3ZIXU5FCDEKDHFP"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2IJK274J79KQFXJ3ZIXU5FCDEKDHFP"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2IJK274J79KQFXJ3ZIXU5FCDEKDHFP"},{"label":"confirm_closed","worker":"A3TXO6RKFIDFUV","object":"234EGOOGQSCMP9S5E65I2UU0A36EEO"},{"label":"confirm_closed","worker":"A3PRF4IIM2IQN2","object":"234EGOOGQSCMP9S5E65I2UU0A36EEO"},{"label":"confirm_closed","worker":"A31X3JCHS0BPFJ","object":"234EGOOGQSCMP9S5E65I2UU0A36EEO"},{"label":"confirm_yes","worker":"A3TXO6RKFIDFUV","object":"23QJIA0RYNJ9LE1FYEWBUJTURDD27P"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"23QJIA0RYNJ9LE1FYEWBUJTURDD27P"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"23QJIA0RYNJ9LE1FYEWBUJTURDD27P"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2S5VP3TVOK5UYANWALHEWZFY07I52D"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2S5VP3TVOK5UYANWALHEWZFY07I52D"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2S5VP3TVOK5UYANWALHEWZFY07I52D"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2ETAT2D5NQYNO6LLCX751YMNMBF84E"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2ETAT2D5NQYNO6LLCX751YMNMBF84E"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2ETAT2D5NQYNO6LLCX751YMNMBF84E"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2Z2MN9U8P9Y3RKER9WUS2YIMAY9EVU"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2Z2MN9U8P9Y3RKER9WUS2YIMAY9EVU"},{"label":"confirm_yes","worker":"A1I3CXC17NIRWB","object":"2Z2MN9U8P9Y3RKER9WUS2YIMAY9EVU"},{"label":"confirm_yes","worker":"ABT7QTMIYXYO0","object":"2GXYQS1CSTMOYO984I9C9EYDXCZPM9"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2GXYQS1CSTMOYO984I9C9EYDXCZPM9"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2GXYQS1CSTMOYO984I9C9EYDXCZPM9"},{"label":"blank","worker":"AH5ZIMHL7TNWC","object":"2BLK4F0OHOVRKV0SW2TLH2HEMBR23K"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2BLK4F0OHOVRKV0SW2TLH2HEMBR23K"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2BLK4F0OHOVRKV0SW2TLH2HEMBR23K"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2X6OGQSCM6IATTA9U8LU80OQUZQHHN"},{"label":"confirm_yes","worker":"A2KET1HL1COET5","object":"2X6OGQSCM6IATTA9U8LU80OQUZQHHN"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2X6OGQSCM6IATTA9U8LU80OQUZQHHN"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"29UWY2AOO2GM55YJ0UHSKM6IKPI442"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"29UWY2AOO2GM55YJ0UHSKM6IKPI442"},{"label":"confirm_yes","worker":"AH5ZIMHL7TNWC","object":"29UWY2AOO2GM55YJ0UHSKM6IKPI442"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2DJVP9746OQ1BE8LJ4X7851QUR3L1Z"},{"label":"confirm_yes","worker":"ABT7QTMIYXYO0","object":"2DJVP9746OQ1BE8LJ4X7851QUR3L1Z"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2DJVP9746OQ1BE8LJ4X7851QUR3L1Z"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"25XXRDS4IC1EH45SVTD19A2I5DKWZS"},{"label":"confirm_no","worker":"A3PRF4IIM2IQN2","object":"25XXRDS4IC1EH45SVTD19A2I5DKWZS"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"25XXRDS4IC1EH45SVTD19A2I5DKWZS"},{"label":"confirm_yes","worker":"A2VRQML8Q3XYP4","object":"2DABRI8IUB2YD0QET6KLJ3L0PQZH56"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2DABRI8IUB2YD0QET6KLJ3L0PQZH56"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2DABRI8IUB2YD0QET6KLJ3L0PQZH56"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2TOSK9RJ85OZ1QWCYO1PUOOQFTCU60"},{"label":"confirm_yes","worker":"AH5ZIMHL7TNWC","object":"2TOSK9RJ85OZ1QWCYO1PUOOQFTCU60"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2TOSK9RJ85OZ1QWCYO1PUOOQFTCU60"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2IBHV8ER9Y8T6B0HB6D5SFHSC334RM"},{"label":"confirm_yes","worker":"ABT7QTMIYXYO0","object":"2IBHV8ER9Y8T6B0HB6D5SFHSC334RM"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2IBHV8ER9Y8T6B0HB6D5SFHSC334RM"},{"label":"confirm_yes","worker":"A3TXO6RKFIDFUV","object":"294EZZ2MIKMNSLQKLCU81WWXSI97O0"},{"label":"confirm_yes","worker":"A19BL9GZGXFWFT","object":"294EZZ2MIKMNSLQKLCU81WWXSI97O0"},{"label":"confirm_yes","worker":"A1I3CXC17NIRWB","object":"294EZZ2MIKMNSLQKLCU81WWXSI97O0"},{"label":"confirm_yes","worker":"A2VRQML8Q3XYP4","object":"2AOYTWX4H3H282M8LN7IEIJRLKM4ZF"}]}' {"timestamp":"2013-05-11T16:03:34.374-07:00","status":"OK","redirect":"responses/3183/POST/jobs/RANDOM__1368313326658_39/assigns"}
Step 3:
$ curl -X GET "http://localhost/troia-server-1.1/responses/3183/POST/jobs/RANDOM__1368313326658_39/assigns" {"timestamp":"2013-05-11T16:04:22.189-07:00","result":"Assigns added","executionTime":0.002,"status":"OK"}
Step 4:
$ curl -X GET "http://localhost/troia-server-1.1/jobs/RANDOM__1368313326658_39/workers/quality/estimated" {"timestamp":"2013-05-11T16:05:21.667-07:00","status":"OK","redirect":"responses/3204/GET/jobs/RANDOM__1368313326658_39/workers/quality/estimated"}
Step 5:
$ curl -X GET "http://localhost/troia-server-1.1/responses/3204/GET/jobs/RANDOM__1368313326658_39/workers/quality/estimated" {"timestamp":"2013-05-11T16:06:01.420-07:00","result":[{"workerName":"A2KET1HL1COET5","value":NaN},{"workerName":"A31X3JCHS0BPFJ","value":NaN},{"workerName":"A3PRF4IIM2IQN2","value":NaN},{"workerName":"A1FQYUBCBNTQX7","value":NaN},{"workerName":"ABT7QTMIYXYO0","value":NaN},{"workerName":"A2L2ZXRHK3SNOP","value":NaN},{"workerName":"A2VRQML8Q3XYP4","value":NaN},{"workerName":"A1R7CJMWXC79UO","value":NaN},{"workerName":"A19BL9GZGXFWFT","value":NaN},{"workerName":"AH5ZIMHL7TNWC","value":NaN},{"workerName":"A1I3CXC17NIRWB","value":NaN},{"workerName":"A3TXO6RKFIDFUV","value":NaN}],"executionTime":0.0,"status":"OK"}
Thanks for reporting this issue! Dana have turned it into test. We will let you know when we manage to fix this.
Quick update:
It looks like there is some problem with dynamic prior calculations done by IDS algorithm. We are still working on this.
Thanks for the update. The workaround is very helpful. I will try to use priors while I am waiting for the fix.
I have implemented the workaround with priors, but I am seeing some strange numbers:
$ curl -X GET "http://localhost/troia-server-1.1/responses/1148/GET/jobs/RANDOM__1368726379654_33/workers/quality/estimated"
{"timestamp":"2013-05-16T10:53:46.686-07:00","result":[{"workerName":"A1W0N2QACQZEQP","value":-3.1727002269058184},{"workerName":"A224TK7J4KA1LV","value":-3.3285464160473195},{"workerName":"A1GU6OYGOV6PBK","value":0.12620888888888937},{"workerName":"AHHKIENAC9M3M","value":0.6288671324385611},{"workerName":"A239WM193Q1OIY","value":-3.3466950270926326}],"executionTime":0.0,"status":"OK"}
Is it normal for worker quality numbers to be negative?
It works that way: quality is computed as 1 - (worker_cost / spammer_cost). workers cost is expected costs that he will generate making an assign. Our baseline is spammer_cost which is using priors and costs to select label which minimizes cost. So in this case spammer is generating lower costs than workers ... So it looks like quality can be negative. Nevertheless, I will ask someone smarter than me about this ;) BTW As you can see changing priors will result in different worker qualities. More accurate priors should result in better worker quality estimation
It is confirmed that in some cases worker quality can be negative. It looks like that this is not enough data for IDS to coverage and it is recommended for such small jobs use BDS algorithm. IDS works well with high volume of assigns. We won't tackle this problem any time soon.
For a number of our jobs, I am getting worker quality as NaN for all workers. Please see comments below for steps to reproduce.