LayoutNet v2

PyTorch implementation of our IJCV paper:

Manhattan Room Layout Reconstruction from a Single 360 image: A Comparative Study of State-of-the-art Methods

https://arxiv.org/pdf/1910.04099.pdf

• New: Our MatterportLayout Annotation dataset is released!

Original Torch implementation for LayoutNet is here.

Improvements upon LayoutNet

• Extend to general Manhattan layout (on our newly labeled MatterportLayout dataset)
• Use ResNet encoder instead of SegNet encoder
• Training details and implementation details
• Gradient ascent based post optimization, revised from sunset1995's PyTorch implementation
• Add random stretching data augmentation

Requirements

• Python 3
• PyTorch >= 0.4.0
• numpy, scipy, pickle, skimage, sklearn, random, cv2, shapely
• torchvision
• Matlab (for depth rendering)

Download Data and Pre-trained Model

• Download pre-trained models and put them under the ./model/ folder.
• Download pre-processed PanoContext and Stanford 2D-3D dataset in .pkl form and put them under the ./data/ folder.
• Download gt from original LayoutNet and the processed one by sunset1995 and put them under the ./data/ folder
• (Optional) Download original LayoutNet's .t7 file and put them under the ./data/ folder
• Download our newly labeled MatterportLayout dataset and put them under the ./data/ folder.
• Download pre-computed depth maps from our models trained on MatterportLayout dataset and put them under the current folder.

Preporcess

• We've provided sample code to transform original LayoutNet's .t7 file to .pkl file for PyTorch

  python t72pkl.py

Training

• On PanoContext (note that we use Stanford 2D-3D as additional data in this script):

  python train_PC.py

• On Stanford 2D-3D (note that we use PanoContext as additional data in this script):

  python train_stanford.py

• On MatterportLayout

  python train_matterport.py

Evaluation

• On PanoContext (Corner error, pixel error and 3D IoU)

  python test_PC.py

• On Stanford 2D-3D (Corner error, pixel error and 3D IoU)

  python test_stanford.py

• On Matterport3D (3D IoU, 2D IoU on under top-down view, RMSE for depth and delta_1 for depth)

  python test_matterport.py

  For depth related evaluation, we need to render depth map from predicted corner position on equirectangualr view (you can skip this step as we've provided pre-computed depth maps from our approach)

  First, uncomment L313-L314 in test_matterport.py, and comment out lines related to evaluation for depth. Run test_matterport.py and save intermediate corner predictions to folder ./result_gen. Then open matlab:

  cd matlab
  cor2depth
  cd ..

  Rendered depth maps will be saved to folder ./result_gen_depth/. Then comment out L313-L314 in test_matterport.py, uncomment lines related to evaluation for depth, and run test_matterport.py again

Citation

Please cite our paper for any purpose of usage.

@article{zou2021manhattan,
  title={Manhattan Room Layout Reconstruction from a Single $ $360\^{}$\{$$\backslash$circ$\}$ $$360∘ Image: A Comparative Study of State-of-the-Art Methods},
  author={Zou, Chuhang and Su, Jheng-Wei and Peng, Chi-Han and Colburn, Alex and Shan, Qi and Wonka, Peter and Chu, Hung-Kuo and Hoiem, Derek},
  journal={International Journal of Computer Vision},
  volume={129},
  number={5},
  pages={1410--1431},
  year={2021},
  publisher={Springer}
}