This repository can be used for reproducing the result we published in our paper. We are constantly working on modeling interactions for road users and planning/prediction-focused simulations. Follow us on Google Scholar for our latest works on these interesting topics.
What you can expect in this repository 🤟:
What you might not find in this repository 😵:
Login and download the dataset from the Waymo Open Dataset. We used the tf.Example proto file from the interactive validation/testing dataset.
As only a subset of scenarios in the training dataset has annotated interactive agents, we provide a script to filter out interactive scenarios from training data. Details are provided below.
Our run.py
script includes the code to compile the Cython script utils_cython.pyx
on each run. In case that does not work out on your machine, use the following command to compile instead.
cd src
cython -a utils_cython.pyx && python setup.py build_ext --inplace
Requires:
Install packages into a Conda environment (Cython, tensorflow, waymo-open-dataset, etc.):
conda env create -f conda.cuda111.yaml
conda activate M2I
Download our prediction results and our pre-trained models from Google Drive, then you can run the following commands for a quick prediction.
Download our relation prediction results m2i.relation.v2v.VAL
on the interactive validation dataset,
or download the ground truth label validation_interactive_gt_relations.pickle
and
the pre-trained relation model m2i.relation.v2v.zip
(unpack first) to run the following command to predict.
Download and unpack to the project folder to load.
OUTPUT_DIR=m2i.relation.v2v; \
DATA_DIR=./validation_interactive/; \
RELATION_GT_DIR=./validation_interactive_gt_relations.pickle; \
python -m src.run --waymo --data_dir ${DATA_DIR} \
--config relation.yaml --output_dir ${OUTPUT_DIR} \
--future_frame_num 80 \
--relation_file_path ${RELATION_GT_DIR} --agent_type vehicle \
--distributed_training 1 -e --nms_threshold 7.2 \
--validation_model 25 --relation_pred_threshold 0.9
Download the marginal prediction results validation_interactive_m2i_v.pickle
on the interactive validation dataset,
or run the following command to predict with the pre-trained marginal prediction model densetnt.raster.vehicle.1.zip
(unpack first):
OUTPUT_DIR=densetnt.raster.vehicle.1; \
DATA_DIR=./validation_interactive/; \
python -m src.run --do_train --waymo --data_dir ${DATA_DIR} \
--output_dir ${OUTPUT_DIR} --hidden_size 128 --train_batch_size 64 \
--sub_graph_batch_size 4096 --core_num 16 \
--other_params l1_loss densetnt goals_2D enhance_global_graph laneGCN point_sub_graph laneGCN-4 stride_10_2 raster train_pair_interest save_rst \
--dist 1 --future_frame_num 80 --agent_type vehicle -e --nms 7.2 --eval_exp_path validation_interactive_v_rdensetnt_full
Download our pre-trained conditional prediction model m2i.conditional.v2v.zip
and unpack it to predict trajectories of
the reactors by running:
OUTPUT_DIR=m2i.conditional.v2v; \
DATA_DIR=./validation_interactive/; \
RELATION_GT_DIR=./validation_interactive_gt_relations.pickle; \
RELATION_PRED_DIR=./m2i.relation.v2v.VAL; \
INFLUENCER_PRED_DIR=./validation_interactive_m2i_v.pickle; \
python -m src.run --waymo --data_dir ${DATA_DIR} \
--output_dir ${OUTPUT_DIR} --config conditional_pred.yaml \
--relation_file_path ${RELATION_GT_DIR} \
--relation_pred_file_path ${RELATION_PRED_DIR} \
--influencer_pred_file_path ${INFLUENCER_PRED_DIR} \
--future_frame_num 80 \
-e --eval_rst_saving_number 0 \
--eval_exp_path ${RESULT_EXPORT_PATH}
The file validation_interactive_gt_relations.pickle
, m2i.relation.v2v.VAL
, validation_interactive_m2i_v.pickle
can be found in the Google drive.
This command will output 6 predictions conditioned on one influencer prediction based on
--eval_rst_saving_number
. Change this variable from 0 to 5 to get 6 groups of conditional predictions.
Results of this pre-trained model on the Waymo Open Motion Dataset interactive prediction benchmark:
Set | Type | minFDE | MR | mAP |
---|---|---|---|---|
Validation (8s) | Vehicle | 5.49 | 0.55 | 0.18 |
Pedstrian | 3.61 | 0.60 | 0.06 | |
Cyclist | 6.26 | 0.73 | 0.04 | |
Test (8s) | Vehicle | 5.65 | 0.57 | 0.16 |
Pedstrian | 3.73 | 0.60 | 0.06 | |
Cyclist | 6.16 | 0.74 | 0.03 |
Waymo Open Dataset does not provide a separate interactive training data partition. We provide a script to filter out interactive data from the training set:
python scripts/filter_interactive_data.py -i TRAINING_DATA_DIR -o TRAINING_INTERACTIVE_DATA_DIR
Download the ground truth relation data training_interactive_gt_relations.pickle
from
Google drive and run the following command to train a relation predictor:
DATA_DIR=./training_interactive/; \
RELATION_GT_DIR=./training_interactive_gt_relations.pickle; \
python -m src.run --do_train --waymo --data_dir ${DATA_DIR} \
--output_dir ${OUTPUT_DIR} --hidden_size 128 --train_batch_size 16 --sub_graph_batch_size 1024 --core_num 16 \
--future_frame_num 80 \
--relation_file_path ${RELATION_GT_DIR} --weight_decay 0.3 --agent_type vehicle \
--other_params train_relation pair_vv \
l1_loss densetnt goals_2D enhance_global_graph laneGCN point_sub_graph laneGCN-4 stride_10_2 raster \
--distributed_training 8
This command trains a relation predictor for vehicle-vehicle interactions.
Replace pair_vv
with pair_vc, pair_vp, pair_others
to train for vehicle-cyclist, vehicle-pedestrian, and the rest type combinations, respectively.
Change the parameter --distributed_training
or --dist
to the number of GPUs you have for training.
Use the following command to train a marginal predictor:
OUTPUT_DIR=waymo.densetnt.raster.1; \
DATA_DIR=./training_interactive/; \
python src/run.py --do_train --waymo --data_dir ${DATA_DIR} \
--output_dir ${OUTPUT_DIR} --hidden_size 128 --train_batch_size 64 --sub_graph_batch_size 4096 --core_num 16 \
--other_params l1_loss densetnt goals_2D enhance_global_graph laneGCN point_sub_graph laneGCN-4 stride_10_2 raster \
--dist 8 --future_frame_num 80 --agent_type vehicle
This command trains a marginal predictor for the vehicle type.
Change the parameter --agent_type
to pedestrian
or cyclist
to train for other types' agents.
Use the following command to train the conditional predictor with the ground truth influencer trajectory and relations.
DATA_DIR=./training_interactive/; \
RELATION_GT_DIR=./training_interactive_gt_relations.pickle; \
python -m src.run --do_train --waymo --data_dir ${DATA_DIR} \
--output_dir ${OUTPUT_DIR} --hidden_size 128 --train_batch_size 64 --sub_graph_batch_size 4096 --core_num 10 \
--future_frame_num 80 --agent_type vehicle \
--relation_file_path ${RELATION_GT_DIR} --weight_decay 0.3 \
--infMLP 8 --other_params train_reactor gt_relation_label gt_influencer_traj pair_vv raster_inf raster \
l1_loss densetnt goals_2D enhance_global_graph laneGCN point_sub_graph laneGCN-4 stride_10_2 \
--distributed_training 8
To filter the type of agents for reactors, change the value of --agent_type
. To filter both the type of reactors and influencers, change the flag of pair_vv
.
Following the policy from Waymo, we might ask you to sign their 'Waymo Dataset License Agreement for Non-Commercial Use' by agreeing to this page before downloading our pre-trained model.
If you found this repo useful to your research, please consider citing
@inproceedings{sun2022m2i,
title={{M2I}: From Factored Marginal Trajectory Prediction to Interactive Prediction},
author={Sun, Qiao and Huang, Xin and Gu, Junru and Williams, Brian and Zhao, Hang},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2022}
}