Possible crowdsourcing strategies for primary study selection

[azhe825]

1 from Wallace

Combining crowd and expert labels using decision theoretic active learning 2015

Strategy:

Two choices each step:

• Pick an unlabeled item and ask the crowd to label it (i.e., collect k crowds labels for a cost of k units).
• Pick a crowd-labeled item and ask the expert to label it for a cost of E units.

Decision theory for which action to choose.

Performance is measured by an expected loss vs cost curve. Not very convincing to me.

2

If crowd worker is way cheaper than expert. (Say 1/1000)

Strategy:

1. ask the crowd to label everything, multiple times (e.g. each item will be labeled by N crowd workers)
2. rank items by the crowd-labeled score, ask experts to review and label items in the ranked order.
3. when enough "relevant" retrieved by experts, start training.
4. re-rank items by a combination of crowd-labeled score and model prediction, ask experts to review and label items in the ranked order. retrain model, repeat 4 until finished.

In 4, can use "true" label (expert label) to adjust weight of crowd labels, e.g. this crowd worker is unreliable...

3

If crowd worker is not that cheap

Strategy:

1. random sample X items, ask crowd to label
2. rank the X items by the crowd-labeled score, ask experts to review and label items in the ranked order.
3. when enough "relevant" retrieved by experts, start training (using only expert labels). Otherwise go to 1.
4. re-rank items by model prediction, ask crowd workers to label top Y, then ask experts to label top Z crowd scored items among the Y. retrain model with expert labels only. repeat 4 until finished.

   Questions

5. how much cheaper does a crowd worker cost comparing to an expert.
6. what to compare? the cost needed to retrieve 90% relevant?
7. how to construct data for experiments.
8. how to design tasks for crowd workers.

[timm]

is this for current paper? or beyond?

[azhe825]

Not for current paper, next step

[azhe825]

also for LN