End-to-end deep learning on real-world 3D data for semantic segmentation and scene captioning.
Paper by Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab and Federico Tombari.
As of 21.03.2019: Best performing geometry-only method on the ScanNet 3D Semantic Labeling Benchmark
a) Textured mesh (not used) b) Input point cloud c) Semantic voxel segmentation
It is recommended to use a virtual environment
git clone https://github.com/drethage/fully-convolutional-point-network.git
./setup.sh
(installs python depdencies and compiles CUDA ops)The project has been tested with Tensorflow 1.12, CUDA 9.0, CUDNN 7.4.1 on Ubuntu 16.04 LTS.
There are three operating modes: training, evaluation and prediction. A JSON-formatted configuration file defines the parameters of an existing or a new session. The structure of this configuration file is described here. Note: this project was originally developed using ScanNet, but is not ScanNet specific. Minimal effort should be required to traing and evaluate this method on new datasets.
Training sessions are differentiated from one another by a session ID defined in config.json
. All artifacts generated during training are placed in a folder corresponding to this session ID in the sessions/
folder.
To start training: python main.py --mode train --config <path_to_config_file>
. Training will begin after the dataset is prepared for training (a cache will be saved to disk).
Alternatively, there is a pre-trained model available in the sessions/
folder for quickly trying out on your own data.
To evaluate a model we present every item in the validation set to the most recent checkpoint present in a session folder, and calculate the intersection-over-union (IoU) over all learnable classes.
To begin evaluation: python main.py --mode evaluate --config <path_to_config_file>
. Note: --config should contain the path to a configuration file in a session folder. Optionally, specify which device to use by setting the --device
flag to cpu or gpu.
After training and evaluating a model, we may wish to predict on unseen data.
To predict on a new mesh or point cloud: python main.py --mode predict --config <path_to_config_file> --file <path_to_PLY_file>
. Optionally, specify which device to use by setting the --device
flag to cpu or gpu. Optionally, use a custom set of colors to represent distinct classes by giving the path to a colors.txt file via --colors
.
If you find our work useful, please consider citing:
@inproceedings{rethage2018eccv,
author = {Rethage, D., Wald, J. Sturm, J. Navab, N., Tombari, F.},
title = {Fully-Convolutional Point Networks for Large-Scale Point Clouds},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2018}
}