This repository contains the source code for our papers:
Deep Patch Visual Odometry
Zachary Teed*, Lahav Lipson*, Jia Deng
Deep Patch Visual SLAM
Lahav Lipson, Zachary Teed, Jia Deng
@article{teed2023deep,
title={Deep Patch Visual Odometry},
author={Teed, Zachary and Lipson, Lahav and Deng, Jia},
journal={Advances in Neural Information Processing Systems},
year={2023}
}
@inproceedings{lipson2024deep,
author={Lipson, Lahav and Teed, Zachary and Deng, Jia},
title={{Deep Patch Visual SLAM}},
booktitle={European Conference on Computer Vision},
year={2024}
}
The code was tested on Ubuntu 20/22 and Cuda 11/12.
Clone the repo
git clone https://github.com/princeton-vl/DPVO.git --recursive
cd DPVO
Create and activate the dpvo anaconda environment
conda env create -f environment.yml
conda activate dpvo
Next install the DPVO package
wget https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.zip
unzip eigen-3.4.0.zip -d thirdparty
# install DPVO
pip install .
# download models and data (~2GB)
./download_models_and_data.sh
Note: You will need to have CUDA 11 and CuDNN installed on your system.
Step 1: Install Pangolin (need the custom version included with the repo)
./Pangolin/scripts/install_prerequisites.sh recommended
mkdir Pangolin/build && cd Pangolin/build
cmake ..
make -j8
sudo make install
cd ../..
Step 2: Install the viewer
pip install ./DPViewer
For installation issues, our Docker Image supports the visualizer.
We provide a classical backend for closing very large loops, which requires extra installation.
Step 1. Install the OpenCV C++ API. On Ubuntu, you can use
sudo apt-get install -y libopencv-dev
Step 2. Install DBoW2
cd DBoW2
mkdir -p build && cd build
cmake .. # tested with cmake 3.22.1 and gcc/cc 11.4.0 on Ubuntu
make # tested with GNU Make 4.3
sudo make install
cd ../..
Step 3. Install the image retrieval
pip install ./DPRetrieval
DPVO can be run on any video or image directory with a single command. Note you will need to have installed DPViewer to visualize the reconstructions in real-time. You can also save the completed reconstructions and view them in COLMAP. The pretrained models can be downloaded from google drive models.zip if you have not already run the download script.
python demo.py \
--imagedir=<path to image directory or video> \
--calib=<path to calibration file> \
--viz # enable visualization
--plot # save trajectory plot
--save_ply # save point cloud as a .ply file
--save_trajectory # save the predicted trajectory as .txt in TUM format
--save_colmap # save point cloud + trajectory in the standard COLMAP text format
python demo.py --imagedir=movies/IMG_0492.MOV --calib=calib/iphone.txt --stride=5 --plot --viz
Download a sequence from TartanAir (several samples are availabe from download directly from the webpage)
python demo.py --imagedir=<path to image_left> --calib=calib/tartan.txt --stride=1 --plot --viz
Download a sequence from EuRoC (download ASL format)
python demo.py --imagedir=<path to mav0/cam0/data/> --calib=calib/euroc.txt --stride=2 --plot --viz
To run DPVO with a SLAM backend (i.e., DPV-SLAM), add
--opts LOOP_CLOSURE True
to any evaluate_X.py
script or to demo.py
If installed, the classical backend can also be enabled using
--opts CLASSIC_LOOP_CLOSURE True
We provide evaluation scripts for TartanAir, EuRoC, TUM-RGBD and ICL-NUIM. Up to date result logs on these datasets can be found in the logs
directory.
Results on the validation split and test set can be obtained with the command:
python evaluate_tartan.py --trials=5 --split=validation --plot --save_trajectory
python evaluate_euroc.py --trials=5 --plot --save_trajectory
python evaluate_tum.py --trials=5 --plot --save_trajectory
python evaluate_icl_nuim.py --trials=5 --plot --save_trajectory
python evaluate_kitti.py --trials=5 --plot --save_trajectory
Make sure you have run ./download_models_and_data.sh
. Your directory structure should look as follows
├── datasets
├── TartanAir.pickle
├── TartanAir
├── abandonedfactory
├── abandonedfactory_night
├── ...
├── westerndesert
...
To train (log files will be written to runs/<your name>
). Model will be run on the validation split every 10k iterations
python train.py --steps=240000 --lr=0.00008 --name=<your name>