Towards High-Fidelity Single-view Holistic Reconstruction of Indoor Scenes (ECCV 2022)

This is the repository of our paper 'Towards High-Fidelity Single-view Holistic Reconstruction of Indoor Scenes' in ECCV 2022
Paper - ArXiv - pdf (abs)
# Environment prerequsite 1. CUDA 11.1 2. pytorch 1.9.0 3. torchvision 0.10.0 after installing the above software, run the following commands for installing others packages: ```angular2html pip install -r requirements.txt ``` # Data Preparation ### Pix3D dataset download prepared pix3d data in the following link: pix3d_data.zip. Unzip the folder and put the pix3d folder under ./data. ### 3D FRONT dataset The prepare data for training is in the following link: training_data. Download prepare_data.zip, occ.zip, mask.zip, and unzip them under ./data/3dfront. **Ps.** The zip file is compressed on **Windows**, it should be fine to unzip it using **winRAR**. If you encounter error during unzip in **Linux**, try to use **7zip** to unzip the file. ### Preprocessing Scripts Prepared watertight model can be downloaded via 3D-FUTURE-watertight.zip.
You can choose to generate watertight model by yourself. First, install Manifold, which can be refered to Manifold. Then, cd into data_preparation folder, run the following commands to convert 3D FUTURE CAD model into watertight model (make sure manifold_root points to the executable file): ```angular2html python prepare_watertight.py --data_root --save_root --manifold_root ``` and run the following commands to sample occupancy for 3D FUTURE dataset. ```angular2html python sample_points.py --data_root --target_root ``` Make sure to install trimesh with embree to accelerate the computation.
Then, download 3d-front-layout.zip. This folder will be used in the later script as layout_root. You can choose to generate your own layout for 3D-FRONT, but you will need to extract the depth image from the prepare_data.zip in the OneDrive Shared Folder, the desc.json will be provided in the raw rendering data 3d-front-raw.zip. The extracted data should be used as **data_root** in the following script. ```angular2html python preprocess_layout.py --data_root --save_root ``` And download 3d-front-raw.zip. This folder will be used in the later script as **data_root**. Run the following commands to preprocess detection data by: ```angular2html python prepare_detection_data.py --data_root --save_root --FRONT3D_root --layout_root ``` FRONT3D_root is a folder extracted from 3D-FRONT.zip, which can be downloaded on the 3D FRONT official website, it contains json files that describe the scene's objects and layout. This script ignore to merge image and depth data, since they are already provided in training_data. ## Object Reconstruction ### Training run the following commands for training: ```angular2html python main.py --mode train --config ./configs/train_instPIFu.yaml ``` After the training is finished, the weight file will be stored in ./checkpoints//model_best.pth. Running the configuraion file ./configs/train_instPIFu_onpix3d.yaml will train instPIFu on pix3d dataset. ### Testing run the following commands to extract mesh result: ```angular2html python main.py --mode test --config ./configs/test_instPIFu.yaml ``` we provide weight file in model_best.pth. you can download it for testing on 3D-FRONT data. The weight entry in the config file is required to be modified to the weight file that you want to test. The mesh files will be saved in ./checkpoints//xxx.ply <\br> weight file for pix3d dataset is in model_best_pix3d.pth. Download the weigt file, and change the weight entry in ./configs/test_instPIFu_onpix3d.yaml, you will be able to inference on pix3d dataset. ### Evaluation For evaluation, gaps is required to be installed for conduct ICP alignment. Run the following commands to install gaps: ```angular2html cd external bash build_gaps.sh ``` Download the all ground truth mesh, and unzip it. run the following commands for evaluation: ```angular2html python evaluate_object_reconstruction.py --result_dir ./checkpoints/ --gt_dir ./Path/to/gt/watertight/mesh ``` evaluation is only conducted on 2000 samples inside ./data/3dfront/split-filter/test.json evaluation results on 3D-FUTURE: | Category | cabinet | chair | table | sofa | bed | night_stand | dresser | desk | bookshelf | Total | |:-----------------|:--------|:------|:------|:-----|:-----|:------------|:--------|:------|:----------|:------| | Chamfer Distance | 5.53 | 10.33 | 17.07 | 5.76 | 7.80 | 18.29 | 40.03 | 52.31 | 3.66 | 10.92 | ## Background reconstruction ### training The occupancy data for background is already updated in bgocc.zip. unzip the folder, and rename the folder as bgocc, then put it under ./data/3dfront/bgocc
run the following commands to train background reconstruction: ```angular2html python main.py --mode train --config ./configs/train_bg_PIFu.yaml ``` run the following commands to extract background mesh: ```angular2html python main.py --mode test --config ./configs/test_bg_PIFu.yaml ``` you can also try to use the pretrained weight at model_best_bg.pth. And modify the weight entry in the configuration file as your weight path
background ground truth depth map is inside training_data as bgdepth.zip, unzip this file and put it under ./data/3dfront run the following commands to evaluate background reconstruction: ```angular2html python evaluate_bg.py --gt_dir ./data/3dfront/bgdepth --result_dir ./checkpoints/xxx ``` ## Object detection Our object detection is based on Im3D. There are three stages for training. I release the pretrained weight on 3D-Front dataset in model_best_det.pth. It is recommended to use this pretrained weight to generate 3D bounding box and camera pose estimation. You can download the pretrained weight, and put it under ./checkpoints/det_weight. Testing is conducted by running the following commands: ```angular2html python main.py --mode test --config ./configs/inference_object_detection.yaml ``` Detection results can be found in detection_result.zip. Unzip the folder, and make sure to modify the pred_pose_path entry in test_instPIFu.yaml to use the detection results for object reconstruction. ### testing object reconstrution with predicted pose you can update the entry use_pred_pose, and pred_pose_path (path storing the object detection result) in test_instPIFu.yaml to use the predicted pose during mesh reconstruction. ## Demo Visualization results can be obtained by running, make sure to run the testid which exists in ./data/3dfront/split/test: ```angular2html python demo.py --testid rendertask6452 ``` The results will be saved into ./output/rendertaskxxx/*.ply.
visualization results will be similar as below(testid is rendertask6452):

### Demo on SUNRGBD dataset Download prepared SUNRGBD data from sunrgbd_train_test_data.zip. Put the sunrgbd_train_test_data folder under ./data/sunrgbd.
We train object reconstruction on Pix3D and test it on SUNRGBD. Change the weight entry in ./configs/demo_sunrgbd_instPIFu.yaml so that it points to model_best_pix3d.pth.
Download SUNRGBD detection results sunrgbd_det_results.zip, which is generated by IM3D. Change the pred_pose_path entry in ./configs/demo_sunrgbd_instPIFu.yaml, so that it points to sunrgbd_det_results/visualization.
run the following command, make sure the testid appears in ./data/sunrgbd/test_objects.json: ```angular2html python demo_sunrgbd --testid 930 ``` visualization results will be as below (testid is 930):

# Citation If you find our work useful, please cite ```angular2 @inproceedings{liu2022towards, title={Towards High-Fidelity Single-view Holistic Reconstruction of Indoor Scenes}, author={Liu, Haolin and Zheng, Yujian and Chen, Guanying and Cui, Shuguang and Han, Xiaoguang}, booktitle={Proceedings of European Conference on Computer Vision}, year={2022} } ```