search

Event-AHU / Mamba_FETrack

[PRCV-2024] State Space Model based Frame-Event Tracking
25 stars 1 forks source link
event-based-tracking fetrack frame-event-tracking mamba mamba-fetrack multi-modal prcv-2024 rgb-event-tracking ssm state-space-model

readme

Mamba_FETrack

News

:dart: Abstract

RGB-Event based tracking is an emerging research topic, focusing on how to effectively integrate heterogeneous multi-modal data (synchronized exposure video frames and asynchronous pulse Event stream). Existing works typically employ Transformer based networks to handle these modalities and achieve decent accuracy through input-level or feature-level fusion on multiple datasets. However, these trackers require significant memory consumption and computational complexity due to the use of self-attention mechanism. This paper proposes a novel RGB-Event tracking framework, Mamba-FETrack, based on the State Space Model (SSM) to achieve high-performance tracking while effectively reducing computational costs and realizing more efficient tracking. Specifically, we adopt two modality-specific Mamba backbone networks to extract the features of RGB frames and Event streams. Then, we also propose to boost the interactive learning between the RGB and Event features using the Mamba network. The fused features will be fed into the tracking head for target object localization. Extensive experiments on FELT and FE108 datasets fully validated the efficiency and effectiveness of our proposed tracker. Specifically, our Mamba-based tracker achieves 43.5/55.6 on the SR/PR metric, while the ViT-S based tracker (OSTrack) obtains 40.0/50.9. The GPU memory cost of ours and ViT-S based tracker is 13.98GB and 15.44GB, which decreased about $9.5\%$. The FLOPs and parameters of ours/ViT-S based OSTrack are 59GB/1076GB and 7MB/60MB, which decreased about $94.5\%$ and $88.3\%$, respectively. We hope this work can bring some new insights to the tracking field and greatly promote the application of the Mamba architecture in tracking.

FETrack Framework

framework

Environment Settings

Download the mamba-1.1.1 and source code and place it in the project path of Mamba_FETrack. Go to source code and install the corresponding environment.

cd mamba-1.1.1
pip install .

Download the casual-conv1d-1.1.3 and source code and place it in the project path of Mamba_FETrack. Go to source code and install the corresponding environment.

cd ..
cd causal-conv1d-1.1.3
pip install .

Download Dataset

Download tracking datasets FELT OR FE108, and put it in ./data.

Download Checkpoint

Download pre-trained and put it under $/pretrained_models.

Download Trained Weights for Model

Download the trained model weights from Mamba_FETrack_ep0050.pth and put it under $/output/checkpoints/train/mamba_fetrack/mamba_fetrack_felt for test directly.

Training and Testing Script

# train
python tracking/train.py --script mamba_fetrack --config mamba_fetrack_felt --save_dir ./output --mode single --nproc_per_node 1 --use_wandb 0
python tracking/train.py --script mamba_fetrack --config mamba_fetrack_fe108 --save_dir ./output --mode single --nproc_per_node 1 --use_wandb 0

# test
python tracking/test.py mamba_fetrack mamba_fetrack_felt --dataset felt --threads 1 --num_gpus 1
python tracking/test.py mamba_fetrack mamba_fetrack_fe108 --dataset fe108 --threads 1 --num_gpus 1

Evaluation Toolkit

Visualization

framework

Experimental Results

Acknowledgment

[OSTrack] [Mamba] [FELT] [CEUTrack] [FE108]

:newspaper: Citation

If you think this paper is helpful, please feel free to leave a star ⭐️ and cite our paper:

@misc{huang2024mambafetrack,
      title={Mamba-FETrack: Frame-Event Tracking via State Space Model}, 
      author={Ju Huang and Shiao Wang and Shuai Wang and Zhe Wu and Xiao Wang and Bo Jiang},
      year={2024},
      eprint={2404.18174},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}