git clone https://github.com/hkchengrex/vos-benchmark
pip install -e vos-benchmark
pip install vos-benchmark
(from the vos-benchmark root directory)
python benchmark.py -g <path to ground-truth directory> -m <path to prediction directory> -n <number of processes, 16 by default>
from vos_benchmark.benchmark import benchmark
# both arguments are passed as a list -- multiple datasets can be specified
benchmark([path to ground-truth directory], [<path to prediction directory>])
See benchmark.py
for an example, and see vos_benchmark/benchmark.py
for the function signature with additional options.
A results.csv
will be saved in the prediction directory.
I built this tool to accelerate evaluations (J&F) on different video object segmentation benchmarks. Previously, davis2017-evaluation is used, which has several limitations:
I have tested this script on DAVIS-16/17 and confirmed that it produces identical results as the official evaluation script.
Image.open
from PIL. Paletted png files and grayscale png should both work.np.unique
. If there are any types of antialiasing, blurring, smoothing, etc., that spawn new pixel values, this will not work.-d
or --do_not_skip_first_and_last_frame
, or passing skip_first_and_last=False
(if used as a package).-s
or --strict
, or passing strict=True
(if used as a package). In the strict mode, an exception would be thrown if the sets of videos do not match.Official DAVIS 2017 evaluation implementation: https://github.com/davisvideochallenge/davis2017-evaluation
BURST benchmark (evaluates HOTA which is not supported here): https://github.com/Ali2500/BURST-benchmark
TrackEval (a powerful tool with more functionalities): https://github.com/JonathonLuiten/TrackEval
Cutie: https://github.com/hkchengrex/Cutie
DEVA: https://github.com/hkchengrex/Tracking-Anything-with-DEVA
XMem: https://github.com/hkchengrex/XMem
STCN: https://github.com/hkchengrex/STCN
MiVOS: https://github.com/hkchengrex/MiVOS
This is part of the accompanying code of DEVA. You can cite this repository as:
@inproceedings{cheng2023tracking,
title={Tracking Anything with Decoupled Video Segmentation},
author={Cheng, Ho Kei and Oh, Seoung Wug and Price, Brian and Schwing, Alexander and Lee, Joon-Young},
booktitle={ICCV},
year={2023}
}