search

davidjurgens / potato

potato: portable text annotation tool
Other
292 stars 49 forks source link

Instance re-distribution after incomplete submissions #100

Open jl3676 opened 3 hours ago

jl3676 commented 3 hours ago

Hi, I recently used potato to collect annotation data on Prolific and I set the labels_per_instance variable to a fixed number. I ran into the issue that incomplete submissions (some workers didn't finish all the instances assigned to them) remained in the instance distribution counts, so after data collection stopped (new workers were told that they came too late and there were no tasks remaining, so they must return their submissions), I didn't get as much data as I wanted per instance sometimes. I wonder if there is a known solution to this problem? Additionally, for the participants who "came too late," being told that could be frustrating because it could feel like a waste of their time (according to some of the messages they sent me).

Jiaxin-Pei commented 3 hours ago

Hi @jl3676, have you tried to connect to your prolific api? Please follow the guide here to set up the automatic task management with prolific. We also have an example project for this!