search

hkchengrex / vos-benchmark

Fast and general video object segmentation evaluation.
MIT License
25 stars 3 forks source link

The j&f values are not correct #2

Closed JawadTawhidi closed 9 months ago

JawadTawhidi commented 9 months ago

Hi, I am disturbing you a lot. But it is because I am still struggling with calculating j and f values.

  1. I cloned your repo using : git clone https://github.com/hkchengrex/vos-benchmark.git
  2. Then I put the davis2017 and DEVA's results on a folder
  3. Then I run this command: python benchmark.py -g davis2017/DAVIS/Annotations_unsupervised/480p -m davis2017/Unsup-DAVIS17-val-DEVA-EntitySeg-semionline -n 16 --do_not_skip_first_and_last_frame

But the result is not correct, it is showing:

Summary: Global score: J&F: 7.7 J: 6.5 F: 8.9 Time taken: 47.31s

  1. Even I run the command like this: python benchmark.py -g davis2017/DAVIS/Annotations_unsupervised/480p -m davis2017/outputs30nov -n 16 --do_not_skip_first_and_last_frame

in this case the folder "output30nov" contains only videos from the validation set. But again the result is the same.

JawadTawhidi commented 9 months ago

If I change the command like this: python benchmark.py -g davis2017/DAVIS/Annotations_unsupervised/480p -m davis2017/outputs30nov -n 16 --do_not_skip_first_and_last_frame -s

It is saying:

We are NOT SKIPPING the evaluation of the first and the last frame (NOT STANDARD for semi-supervised video object segmentation). Videos that are in davis2017/DAVIS/Annotations_unsupervised/480p but not in davis2017/outputs30nov: {'disc-jockey', 'lady-running', 'scooter-gray', 'bus', 'mallard-fly', 'swing', 'varanus-cage', 'sheep', 'elephant', 'stroller', 'tuk-tuk', 'classic-car', 'longboard', 'flamingo', 'rollerblade', 'walking', 'horsejump-low', 'stunt', 'motocross-bumps', 'tractor-sand', 'car-turn', 'scooter-board', 'surf', 'rhino', 'motorbike', 'bmx-bumps', 'dancing', 'drift-turn', 'kid-football', 'drone', 'train', 'snowboard', 'dogs-scale', 'cat-girl', 'kite-walk', 'miami-surf', 'mallard-water', 'lucia', 'lindy-hop', 'bear', 'hike', 'upside-down', 'tennis', 'hockey', 'color-run', 'night-race', 'boat', 'breakdance-flare', 'planes-water', 'dance-jump', 'paragliding', 'koala', 'dog-agility', 'rallye', 'skate-park', 'crossing', 'schoolgirls', 'dog-gooses', 'soccerball', 'boxing-fisheye'} Validation failed. Exiting.

hkchengrex commented 9 months ago

This script works for DAVIS 2016/2017 semi-supervised or DAVIS 2016 unsupervised. It does not work for DAVIS 2017 unsupervised.