This is an official release of the CVPR 2024 paper: SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking.
Models & Raw Results(Google Driver) Models & Raw Results(Baidu Driver: qolo)
[Mar 26, 2024]
✅ We release codes, models and raw results.
[Feb 27, 2024]
✅ SDSTrack is accepted to CVPR2024.
💡 SDSTrack uses lightweight adapters for parameter-efficient fine-tuning and can be used for various modality combinations.
💡 We employ a symmetric design, treating all modalities equally, thereby reducing modality dependencies.
💡 We significantly improve the robustness of trackers through complementary masked patch distillation.
💡 SDSTrack performs excellently even in extreme scenarios, such as modality absence and severe occlusion.
Create and activate a conda environment:
conda create -n sdstrack python=3.8
conda activate sdstrack
Install the required packages:
bash install_sdstrack.sh
Put the training datasets in ./data/
. It should look like:
$<PROJECT_ROOT>
-- data
-- depthtrack/train
|-- adapter02_indoor
|-- bag03_indoor
...
-- lasher/trainingset
|-- 1boygo
|-- 1handsth
...
-- visevent/train
|-- 00142_tank_outdoor2
|-- 00143_tank_outdoor2
...
|-- trainlist.txt
Run the following command to set paths:
cd <PROJECT_ROOT>
python tracking/create_default_local_file.py --workspace_dir . --data_dir ./data --save_dir ./output
You can also modify paths with these two files if needed:
lib/train/admin/local.py # paths for training
lib/test/evaluation/local.py # paths for testing
Download the pretrained foundation model (OSTrack) and put it under $PROJECT_ROOT$/pretrained
. To train RGB-Depth tracking, you can run:
bash train_sdstrack_rgbd.sh
You can also train various modality combinations by bash train_sdstrack_rgbe.sh
and bash train_sdstrack_rgbt.sh
.
RGB-Depth benchmarks
Two benchmarks (DepthTrack testing set and VOT22_RGBD) are evaluated using VOT-toolkit.
./Depthtrack_workspace/
and rename it sequences
../VOT22RGBD_workspace/
and also rename it sequences
.bash eval_rgbd.sh
RGB-Thermal benchmarks
Two benchmarks (LasHeR testing set and RGBT234) are evaluated.
bash eval_rgbt.sh
We refer you to LasHeR Toolkit for LasHeR evaluation and MPR_MSR_Evaluation for RGBT234 evaluation.
RGB-Event benchmarks
One benchmark (VisEvent) is evaluated. Please modify the ./RGBE_workspace/test_rgbe_mgpus.py
, then run:
bash eval_rgbe.sh
If our work is useful for your research, please consider citing:
@InProceedings{Hou_2024_CVPR,
author = {Hou, Xiaojun and Xing, Jiazheng and Qian, Yijie and Guo, Yaowei and Xin, Shuo and Chen, Junhao and Tang, Kai and Wang, Mengmeng and Jiang, Zhengkai and Liu, Liang and Liu, Yong},
title = {SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2024},
pages = {26551-26561}
}