Predicting Future Occupancy Grids in Dynamic Environment with Spatio-Temporal Learning

Paper | Project website | Video

Abstract

Reliably predicting future occupancy of highly dynamic urban environments is an important precursor for safe autonomous navigation. Common challenges in the prediction include forecasting the relative position of other vehicles, modelling the dynamics of vehicles subjected to different traffic conditions, and vanishing surrounding objects. To tackle these challenges, we propose a spatio-temporal prediction network pipeline that takes the past information from the environment and semantic labels separately for generating future occupancy predictions. Compared to the current SOTA, our approach predicts occupancy for a longer horizon of 3 seconds and in a relatively complex environment from the nuScenes dataset. Our experimental results demonstrate the ability of spatio-temporal networks to understand scene dynamics without the need for HD-Maps and explicit modeling dynamic objects. We publicly release our occupancy grid dataset based on nuScenes to support further research.

Generating Occupancy grids dataset

Occupancy grid maps are generated from the nuScenes dataset. Agents belonging to the ’Vehicles’ category are of interest and being marked by ’green’ semantic pixel labels using the projections of ground truth 3D bounding boxes. Objects of any other type including the static environment are marked in ’blue’. Different road crossing motion scenarios from the dataset are presented below:

We publicly release the Occupancy Grid Maps dataset consisting of static environment and semantic labels for ease in long-term prediction. The dataset is available here The paper demonstrate the performance of two state-of-art video prediction networks - PredRNN and ConvLSTM for this dataset. Under the training scheme we observe that the models have consistent performance even for long-term predictions.

Spatio-temporal architecture

We present an Spatio-Temporal Network Pipeline for long-term future occupancy grid prediction. Our approach uses semantic labels of the vehicle in OGMs to model the specific motion type of agents in the prediction rather than using a generic combined prediction of static and dynamic cells that contains the environment and various types of agents such as cars, pedestrians, cyclists, etc. Semantic occupancy grids consisting of environment and vehicles over the time frames of 0.5 sec. Grids are converted into binary images and separately fed to spatio-temporal networks. We evaluate two spatio-temporal networks: PredRNN and ConvLSTM

Qualitative Results

A scene depicting few static vehicles and a vehicle going in reverse direction. Comparison between two spatio-temporal learning networks over the future predictions of 1 sec, 2 sec and 3 sec.

(a) Ground truth

(b) PredRNN predicitons

(c) ConvLSTM predictions

Quantitative Results

Frame-wise PSNR(↑), SSIM(↑), Static MSE (↓), and Semantic MSE (↓) results on the generated Occupancy grid map dataset are presented below. The prediction horizon is 3 sec during training and testing phases. Note that PredRNN (combined) and ConvLSTM (combined) predict an entire OGM, thus separate Static and Semantic MSE cannot be reported for these cases. The Semantic MSE result from the linear projection of vehicle’s bounding boxes is also presented in (d).

Bibtex

@article{mann2022predicting,
  title={Predicting Future Occupancy Grids in Dynamic Environment with Spatio-Temporal Learning},
  author={Mann, Khushdeep Singh and Tomy, Abhishek and Paigwar, Anshul and Renzaglia, Alessandro and Laugier, Christian},  
  journal={arXiv preprint arXiv:2205.03212},  
  year={2022}
}