Created by Yu Xiang and Tanner Schmidt at RSE-Lab at University of Washington.
We introduce Data Associated Recurrent Neural Networks (DA-RNNs), a novel framework for joint 3D scene mapping and semantic labeling. DA-RNNs use a new recurrent neural network architecture for semantic labeling on RGB-D videos. The output of the network is integrated with mapping techniques such as KinectFusion in order to inject semantic information into the reconstructed 3D scene. arXiv, Video
DA-RNN is released under the MIT License (refer to the LICENSE file for details).
If you find DA-RNN useful in your research, please consider citing:
@inproceedings{xiang2017darnn,
Author = {Yu Xiang and Dieter Fox},
Title = {DA-RNN: Semantic Mapping with Data Associated Recurrent Neural Networks},
Booktitle = {Robotics: Science and Systems (RSS)},
Year = {2017}
}
DA-RNN consists a reccurent neural network for semantic labeling on RGB-D videos and the KinectFusion module for 3D reconstruction. The RNN and KinectFusion communicate via a Python interface.
Install TensorFlow. I suggest to use the Virtualenv installation.
Compile the new layers under $ROOT/lib we introduce in DA-RNN.
cd $ROOT/lib
sh make.sh
Compile KinectFusion with cmake. Unfortunately, this step requires some effort.
Install dependencies of KinectFusion:
cd $ROOT/lib/kinect_fusion
mkdir build
cd build
cmake ..
make
Compile the Cython interface for RNN and KinectFusion
cd $ROOT/lib
python setup.py build_ext --inplace
Add the KinectFusion libary path
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ROOT/lib/kinect_fusion/build
Download the VGG16 weights from here (57M). Put the weight file vgg16_convs.npy to $ROOT/data/imagenet_models.
Download the RGB-D Scene dataset from here (5.5G).
Create a symlink for the RGB-D Scene dataset
cd $ROOT/data/RGBDScene
ln -s $RGB-D_scene_data data
Training and testing on the RGB-D Scene dataset
cd $ROOT
# train and test RNN with different input (color, depth, normal and rgbd)
./experiments/scripts/rgbd_scene_multi_*.sh $GPU_ID
# train and test FCN with different input (color, depth, normal and rgbd)
./experiments/scripts/rgbd_scene_single_*.sh $GPU_ID
Download the ShapeNet Scene dataset from here (2.3G).
Create a symlink for the ShapeNet Scene dataset
cd $ROOT/data/ShapeNetScene
ln -s $ShapeNet_scene_data data
Training and testing on the RGB-D Scene dataset
cd $ROOT
# train and test RNN with different input (color, depth, normal and rgbd)
./experiments/scripts/shapenet_scene_multi_*.sh $GPU_ID
# train and test FCN with different input (color, depth, normal and rgbd)
./experiments/scripts/shapenet_scene_single_*.sh $GPU_ID
You can download all our trained tensorflow models on the RGB-D Scene dataset and the ShapeNet Scene dataset from here (3.1G).
# an exmaple to test the trained model
./experiments/scripts/rgbd_scene_multi_rgbd_test.sh $GPU_ID