search

1712n / challenge

Challenge Program
64 stars 26 forks source link

QA - Identify bots amongst annotators #77

Closed alinapark closed 1 year ago

alinapark commented 2 years ago

While crowdsourcing the annotation of our classification datasets, we have seen a number of bots taking up the tasks. We have asked several MTurk people to annotate texts with a true/false value depending on whether the texts are relevant to a topic of Hacker Attack. There's no way to identify the untrustworthy annotators except through their responses. The goal of this issue is to filter out those bots.

The Challenge

To participate in the challenge, fork the challenge repository, analyze annotators' responses, create a pull request in your repository with all suspicious annotators removed, and assign this issue assignee as the pull request reviewer. Expanding the pull request description with your methodology can help us better understand your reasoning and evaluate your submission faster. To make sure your submission doesn't get lost, you can also email your pull request link along with your resume and the link to this challenge to challenge-submission@blockshop.org. Also, don't hesitate to ask us questions by commenting in this issue.

geoburdin commented 2 years ago

You can check if there is anything useful https://colab.research.google.com/drive/1wyLwzRbjzek4-Vg7iZuNxvMJ728caGLa?usp=sharing

SvetlanaPek commented 2 years ago

Hello! I sent you an email from svetlana.pekarskikh@gmail.com with a proposed solution. Have a nice day!

lidiaToropova commented 2 years ago

Hi! Sent my solution via email (lydiatoropova@gmail.com). Thanks!

mamintatarin commented 2 years ago

Hi! Asked you a question via email hamitov.tm@phystech.edu

achilleess commented 2 years ago

Hello! Sent my solution via email rozhnevn@gmail.com.

holyfameeee commented 1 year ago

Hello! Sent my solution via email (holyyyfaaame@gmail.com) :)

t0pdog commented 1 year ago

Hi! Sent my solution via email shirikovdmitry@gmail.com. Thanks!