Mechanical Turk Integration

cvat-ai / cvat

Annotate better with CVAT, the industry-leading data engine for machine learning. Used and trusted by teams at any scale, for data of any scale.

https://cvat.ai

MIT License

12.72k stars 3.02k forks source link

Mechanical Turk Integration #20

Open la61nn0vate opened 6 years ago

la61nn0vate commented 6 years ago

Integration of CVAT with MTurk for deploying work as HITs would be very useful for such projects. Need to integrate Turkic framework of VATIC with CVAT. I would also like to contribute to your project. Please help me in setting up the development environment for the same.

nmanovic commented 6 years ago

Hi,

It is a feature in our mind which we don't have resources to implement till EOY for sure. If you can help it will be awesome. Development environment is described here: https://github.com/opencv/cvat/issues/18.

Several notes:

Need to implement a separate Django application for the feature
It isn't good to use turkic from VATIC because it isn't supported anymore. I would suggest using an official API (boto3?) - http://boto3.readthedocs.io/en/latest/reference/services/mturk.html
Need to think how split a task on many HITs (by frame/image?) because HITs should be trivial. Also need to automatically create "verification" tasks to check results. In many case it will be necessary to merge annotation from several HITs.
GUI to publish, view progress, ... Need to have sketch of it.
The application should have REST API.

In general it is a big feature which cannot be easily implemented.

nmanovic commented 6 years ago

General requirements:

Easy interface to create/review/accept an annotation task for MTurk (separate Django app).
Probably need to have a separate instance on AWS for such tasks.
Automatic authorization from MTurk, special group to access only links from MTurk django app
Email notifications about deadlines to verify annotation tasks, etc...

Requirements (image annotation):

One image per a segment. Thus it is easy to post such job as HIT.
Several jobs for the same segment.
Automatically correlate annotations in different jobs for the same segment. Automatically find "trash" in annotations and discard such HIT.
Minimum number of jobs for a segment is 3.
Maximum number of jobs for a segment is 7. After that need to verify or discard HITs due to low quality.
One user cannot do annotations for the same segment (several similar jobs). Need to improve auth/permissions to access only specific jobs.

nik13 commented 4 years ago

Hi, Wanted to know if any progress has been made in this direction :)

nmanovic commented 4 years ago

@nik13 , we are not working in the direction at the moment.

nmanovic commented 3 years ago

Nice overview of using a crowd sourcing platform (Toloka): https://www.youtube.com/watch?v=yyybisXHFc0

serwansj commented 1 year ago

Has there been any progress on the integration with crowdsourcing platforms?

nmanovic commented 1 year ago

I hope to have some news in Q2-Q3 on the topic...

eliabruni commented 1 year ago

Very interested in this too!