RaTrack: Moving Object Detection and Tracking with 4D Radar Point Cloud
Zhijun Panβ
Fangqiang Dingβ
Hantao Zhongβ
Chris Xiaoxuan Lu*
β Equal Contribution *Corresponding author
Royal College of Art University of Edinburgh University of Cambridge
University College London
Figure 1. Overall network pipeline of RaTrack. Given each 4D radar point cloud from the stream, we first apply a backbone network to encode intra- and inter-frame radar point cloud features. With the extracted features, our point-wise motion estimation module infers point-level scene flow as an explicit complement to augment the latent features of radar point clouds. Our advocated idea of class-agnostic detection without bounding boxes is introduced in the object detection module in which moving points are first identified and then used to detect moving objects via clustering. Finally, our data association module computes the affinity matrix with a learnable distance metric and then optimises the bipartite matching problem. The entire network is end-to-end trainable with a multi-task loss that incorporates three supervised subtasks: motion segmentation, scene flow estimation, and affinity matrix computation. |
If you find our work useful in your research, please consider citing:
@INPROCEEDINGS{10610368,
author={Pan, Zhijun and Ding, Fangqiang and Zhong, Hantao and Lu, Chris Xiaoxuan},
booktitle={2024 IEEE International Conference on Robotics and Automation (ICRA)},
title={RaTrack: Moving Object Detection and Tracking with 4D Radar Point Cloud},
year={2024},
volume={},
number={},
pages={4480-4487},
keywords={Computer vision;Three-dimensional displays;Motion segmentation;Noise;Radar detection;Radar;Radar imaging},
doi={10.1109/ICRA57147.2024.10610368}}
Here are some GIFs to show our qualitative results on moving object detection and tracking based on 4D radar point clouds. Note that only moving objects with no less than five points. For more qualitative results, please refer to our demo video.
First, please request and download the View of Delft (VoD) dataset from the VoD official website. Unzip into the folder you prefer.
Please also obtain the tracking annotation from VoD Github. Unzip all the .txt
tracking annotation files into the path: PATH_TO_VOD_DATASET/view_of_delft_PUBLIC/lidar/training/label_2_tracking/
The dataset folder structure should look like this:
view_of_delft_PUBLIC/
βββ lidar
βΒ Β βββ ImageSets
βΒ Β βββ testing
βΒ Β βββ training
βΒ Β βββ calib
βΒ Β βββ image_2
βΒ Β βββ label_2
βΒ Β βββ 00000.txt
βΒ Β βββ 00001.txt
βΒ Β βββ ...
βΒ Β βββ label_2_tracking
βΒ Β βββ 00000.txt
βΒ Β βββ 00001.txt
βΒ Β βββ ...
βΒ Β βββ pose
βΒ Β βββ velodyne
βββ radar
βΒ Β βββ testing
βΒ Β βββ training
βΒ Β βββ calib
βΒ Β βββ velodyne
βββ radar_3frames
βΒ Β βββ testing
βΒ Β βββ training
βΒ Β βββ velodyne
βββ radar_5frames
βββ testing
βββ training
βββ velodyne
Please ensure you running with an Ubuntu machine with Nvidia GPU (at least 2GB VRAM). The code is tested with Ubuntu 22.04, and CUDA 11.8 with RTX 4090. Any other machine is not guaranteed to work.
To start, please ensure you have miniconda installed by following the official instructions here.
First, clone the repository with the following command and navigate to the root directory of the project:
git clone git@github.com:LJacksonPan/RaTrack.git
cd RaTrack
Create a RaTrack environment with the following command:
conda env create -f environment.yml
This will setup a conda environment named RaTrack
with CUDA 11.8, PyTorch2.2.0.
Installing the pointnet2 pytorch dependencies:
cd lib
python setup.py install
To train the model, please run:
python main.py
This will use the configuration file config.yaml
to train the model.
To evaluate the model and generate the model predictions, please run:
python main.py --config configs_eval.yaml
To evaluate with the trained RaTrack model, please open the configs_eval.yaml
and change the model_path
to the path of the trained model.
model_path: 'checkpoint/track4d_radar/models/model.last.t7'
Then run the following command:
python main.py --config configs_eval.yaml
This will only generate the predictions in the results
folder. We are currently working on integrating our point-based version of AB3DMOT evaluation scripts into the evaluation run.
The evaluation results of the provided trained RaTrack model are following:
Method | SAMOTA | AMOTA | AMOTP | MOTA | MODA | MT | ML |
---|---|---|---|---|---|---|---|
RaTrack | 74.16 | 31.50 | 60.17 | 67.27 | 77.83 | 42.65 | 14.71 |
RaTrack (Improved) | 80.33 | 34.58 | 59.37 | 62.80 | 77.07 | 54.41 | 13.24 |
Q: Regarding Our Modified AB3DMOT Evaluation Code
A: Due to AB3DMOT's repository license, we are currently not able to distribute our modified version of the AB3DMOT evaluation. We are contacting AB3DMOT's authors to gain permission for distribution.
Q: Why DBSCAN?
A: Due to our time limits, we were not able to implement and integrate a differentiable clustering method into our pipeline. However, all local and global features are utilised and passed through during the clustering process. Adding a mindfully selected and implemented differentiable clustering method could be a potential future work to further improve the pipeline's performance.
We use the following open-source projects in our work: