OpenTrafficCam3D

Toward Planet-Wide Traffic Camera Calibration
Khiem Vuong, Robert Tamburo, Srinivasa G. Narasimhan
IEEE Winter Conference on Applications of Computer Vision (WACV), 2024

[arXiv] [Project Page] [Bibtex]

teaser

Introduction

You have access to a traffic camera stream but want to know its intrinsics and/or extrinsics? OpenTrafficCam3D is a pipeline that can help you calibrate traffic cameras using only the traffic camera image and a few GPS-tagged images (e.g., Google Street View, for metric scale).

The pipeline consists of three main steps: scene reconstruction, traffic camera localization, and geo-registration/ground plane fitting.

Prepare Environment

Clone the repository

git clone https://github.com/kvuong2711/OpenTrafficCam3D.git
cd OpenTrafficCam3D

Build Docker

Either build the docker: sudo docker build . -t kvuong2711/otc3d:latest

or pull the pre-built docker: sudo docker pull kvuong2711/otc3d:latest

Run the docker

sudo docker run --gpus 'all' -v /:/mars --shm-size=64GB -it kvuong2711/otc3d:latest

where / is the directory in the local machine (in this case, the root folder), and /mars is the reflection of that directory in the docker.

Inside the docker, change the working directory to this repository:

cd /mars/PATH/TO/THIS/REPO/OpenTrafficCam3D

Additional Dependencies

Assuming you are at OpenTrafficCam3D directory, we want to install detectron2 and Mask2Former:

# Install detectron2
cd trafficreloc/third_party/detectron2/
pip install .
pip install git+https://github.com/cocodataset/panopticapi.git
pip install git+https://github.com/mcordts/cityscapesScripts.git

# Install Mask2Former
cd ../Mask2Former/
pip install -r requirements.txt
cd mask2former/modeling/pixel_decoder/ops/
sh make.sh
cd ../../../../../../..

After these steps, you should be back at the OpenTrafficCam3D directory.

Data Preparation and Hierarchy

Here, we show how to prepare a small sample set of data for Jackson Hole, WY. To modify the location/extent/number of panoramas and where the data is saved, please modify the locations and default parameters in trafficreloc/data/gsv/prepare.py. By default, the data will be saved in ./sample_data.

You should obtain the API_KEY from Google Maps API.

Run the following command to download the data:

  cd trafficreloc/data/gsv
  python prepare.py --location jhts --api_key YOUR_API_KEY
  cd ../../../

The data will be saved in ./sample_data. The hierarchy of the data for location jhts is as follows:

  sample_data
  |
  ├── raw_data
  │   ├── jhts
  │   │   ├── pano_data  # panorama images downloaded from GSV
  │   │   ├── perspective_data  # extracted perspective images from panos
  |
  ├── processed_data
  │   ├── jhts
  │   │   ├── database
  │   │   |  ├── images  # images for scene reconstruction
  |   |   |
  │   │   ├── query
  │   │   |  ├── images  # images for localization (e.g., traffic cam)

In short, you should put images you want to use for scene reconstruction in ./sample_data/processed_data/jhts/database/images and any images you want to use for localization in ./sample_data/processed_data/jhts/query/images.

If you just want to quickly try the code out without setting up the data downloading pipeline, you can download the sample data from here and extract it to the root directory of the repository (i.e., OpenTrafficCam3D/sample_data).

Disclaimer: The data provided in the sample data is for demonstration purposes only and not to be used for commercial purposes. We do not own the data and we do not distribute it. Please refer to the Google Maps Platform Terms of Service for more information.

Scene Reconstruction

You can modify the data paths and parameters in the configuration file gsv_loc_jhts_release.yml to suit your needs.

Then, run the following command to reconstruct the scene:

python trafficreloc/pipeline_reconstruction.py --config_file trafficreloc/configs/gsv_loc_jhts_release.yml

Traffic Camera Localization

Assuming your data is prepared and the scene has been reconstructed, you can now localize traffic cameras in the scene. You can modify the data paths and parameters in the configuration file gsv_loc_jhts_release.yml to suit your needs.

Then, run the following command to localize traffic cameras:

python trafficreloc/pipeline_localization.py --config_file trafficreloc/configs/gsv_loc_jhts_release.yml

Geo-Registration and Ground Plane Fitting

As a post-processing step, you can geo-register the scene (so that the scene is in metric scale) based on the GPS of the database images and (optionally) fit a ground plane to the scene.

Run the following command to geo-register the scene and fit a ground plane:

python trafficreloc/pipeline_postprocess.py --config_file trafficreloc/configs/gsv_loc_jhts_release.yml --api_key YOUR_API_KEY

You can find all the results in ./sample_data/processed_data/jhts/database/recon_output/sfm_superpoint+superglue, containing the latest_metric_model and groundplane_equation.txt (in COLMAP global coordinate system).

Note:

This step requires the API_KEY to query the Google Maps API for the GPS coordinates of the database images. Optionally, you can provide your own GPS coordinates by modifying the extract_wgs84_txtfile function in trafficreloc/georef_colmap.py.
This code also visualizes the scene using Open3D which usually requires a GUI.

Reference

If you use find this code helpful, please cite:

@inproceedings{vuong2024trafficcalib,
      title = {Toward Planet-Wide Traffic Camera Calibration},
      author = {Vuong, Khiem and Tamburo, Robert and Narasimhan, Srinivasa G.},
      booktitle = {IEEE Winter Conference on Applications of Computer Vision},
      year = {2024},
}

Acknowledgements

We would like to thank the authors of hloc with their codebase as a starting point for our project.

Contact

If you have any questions, please feel free to open an issue or contact Khiem Vuong.

kvuong2711 / OpenTrafficCam3D

readme