search

ipeirotis / Get-Another-Label

Quality control code for estimating the quality of the workers in crowdsourcing environments
70 stars 26 forks source link

"MisclassificationCost" doesn't seem to work #24

Closed danithaca closed 12 years ago

danithaca commented 12 years ago

From both the paper and the code, it seems that we can set MisclassificationCost to be an arbitrary number other than 0 or 1. However, when I set "porn notporn" to be 1000 or 0.1, the results don't change at all. I also set all 4 cases (porn-porn, notporn-notporn, porn-notporn, notporn-porn) to be 0, but the results still don't change. I have a larger dataset, and no matter how I change MisclassificationCost to be different numbers across different categories, the results remain the same. Therefore, I suspect "MisclassificationCost" might have a bug.

Is there a quick answer to this issue? In the meantime, I'll try to do some debugging work too.

ipeirotis commented 12 years ago

Do you have an example so that we can test?

danithaca commented 12 years ago

The input data is at: https://s3.amazonaws.com/balance_default/public/input.tar.gz

I used the following commands to generate results: 1) java -jar get-another-label-2.2.0-SNAPSHOT-jar-with-dependencies.jar category.txt input.txt correct.txt cost1.txt evaluation.txt 2) java -jar get-another-label-2.2.0-SNAPSHOT-jar-with-dependencies.jar category.txt input.txt correct.txt cost2.txt evaluation.txt 3) java -jar get-another-label-2.2.0-SNAPSHOT-jar-with-dependencies.jar category.txt input.txt correct.txt cost3.txt evaluation.txt

Note that they are only different with the cost files: cost1.txt, cost2.txt, cost3.txt.

The results (only object-probabilities.txt) are at: https://s3.amazonaws.com/balance_default/public/result.tar.gz

Note that the columns with the "DS" prefix all have the same values across r1.txt, r2.txt, and r3.txt. The columns with the "MV" prefix have different values. I think the "DS" columns refer to results from DawidSkene, and the values should be different from different cost files. The "MV" columns seem to be reasonable.

Thanks.

ipeirotis commented 12 years ago

I think you have some things mixed up.

First of all, the probabilities should not change with different costs structures (and from what I see they are the same everywhere -- thankfully).

What does change with different costs is the estimation of the DS_Exp_Cost, DS_Opt_Cost, MV_Exp_Cost, etc., which are the estimations of the misclassification costs.

You may want to pass an evaluation file as well with (some of) the correct answers, to see how the different algorithms perform. Compiling the latest source files will also give you the ability to use some of the latest code that reports some additional methods of classification and some additional metrics.

danithaca commented 12 years ago

Thanks! I'll go back and re-read the paper. I thought different cost structures will change the probabilities. My intuition was that suppose misclassification cost for "porn-notporn" was 1000 but for "notporn-porn" was 1, then a worker who always messed with the "porn-notporn" cases would have even lower quality score, and therefore caused object-probabilities to change too. But apparently my intuition was wrong. Hope other people will not have this misunderstanding.

ipeirotis commented 12 years ago

The worker will have a lower quality score.

Regarding object probabilities, in the latest version of the code, we also include the "MinCost" classification as well: The object probabilities remain the same but the object gets classified into the category that generates the lowest expected misclassification cost.

If it is not in the output already, it should be in the next couple of days. (It is Issue #17)

danithaca commented 12 years ago

thanks.