cvondrick / vatic

Efficiently Scaling Up Video Annotation with Crowdsourced Marketplaces. IJCV 2012
http://mit.edu/vondrick/vatic/
MIT License
603 stars 255 forks source link

"Gold Standard Training" video not showing up for new turkers #13

Open forresti opened 9 years ago

forresti commented 9 years ago

I set up a "worker training" video:

turkic load train_video /path/train_video.mp4 Car
    --blow-radius 1 
    --for-training 
    --for-training-start 0 
    --for-training-stop 300
    --for-training-overlap 70 
    --for-training-tolerance 70 
    --for-training-mistakes 1

...and I annotated train_video in the UI.

Then, I set up an ordinary video for a worker to annotate... and notice that I do --train-with train_video:

turkic load new_video /path/new_video.mp4  Car 
    --length 300
    --blow-radius 1 
    --title "Label the cars in this video (15sec of video). Per-object bonus." 
    --description "Draw boxes around the cars in this video. When the cars move, you move the boxes. See 'show instructions' in the HIT. Feel free to email us with questions." 
    --duration 600 
    --lifetime 604800 
    --keywords "video, tagging, annotation, cars" 
    --cost 0.05 
    --per-object-bonus 0.02 
    --completion-bonus 0.1
    --train-with training_video

In the worker sandbox, I tried 2 things:

  1. Login as myself and look at the HIT. Instead of seeing train_video, I immediately see new_video.
  2. Perhaps I've annotated my own tasks in the worker sandbox before. So, I created a new worker account. With my new account, I still immediately see new_video instead of train_video.
  3. I tried using mturk.com instead of the worker sandbox. The training video does show up here. But, the training video appears with my ground truth annotations on it! This can't be right...

Questions

gnetscher commented 8 years ago

I had the same issue and would love to see a response

pammirato commented 8 years ago

did you accept the hit?

For me, if I just look at the hit it shows the new video, but once I accept the hit it starts with the training video

that being said I do need to find a way to "untrain" myself so I can test the training video multiple times in the sandbox