search

monniert / unicorn

(ECCV 2022) Code for Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency
https://www.tmonnier.com/UNICORN
MIT License
162 stars 15 forks source link
3d-reconstruction analysis-by-synthesis computer-vision deep-learning differentiable-rendering eccv-2022 eccv2022 mesh-generation pytorch single-view-reconstruction unsupervised unsupervised-learning

readme

UNICORN :unicorn:

Share With Thy Neighbors:
Single-View Reconstruction by Cross-Instance Consistency

Tom Monnier  Matthew Fisher  Alexei A. Efros  Mathieu Aubry

![teaser.gif](./media/teaser.gif)

Official PyTorch implementation of Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency (ECCV 2022). Check out our webpage for video results!

This repository contains:

If you find this code useful, don't forget to star the repo :star: and cite the paper :point_down: ``` @inproceedings{monnier2022unicorn, title={{Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency}}, author={Monnier, Tom and Fisher, Matthew and Efros, Alexei A and Aubry, Mathieu}, booktitle={{ECCV}}, year={2022}, } ```
Major code updates :clipboard: - 08/22: pytorch 1.10 instead of 1.5, big models, kp_eval.py, eval with gradient-based ICP with aniso scale, pascal3D car chamfer eval - 05/22: first code release

Installation :construction_worker:

1. Create conda environment :wrench:

conda env create -f environment.yml
conda activate unicorn
Optional visualization :chart_with_downwards_trend: Some monitoring routines are implemented, you can use them by specifying your visdom port in the config file. You will need to install visdom from source beforehand: ``` git clone https://github.com/facebookresearch/visdom cd visdom && pip install -e . ```

2. Download datasets :arrow_down:

bash scripts/download_data.sh

This command will download one of the following datasets:

3. Download pretrained models :arrow_down:

bash scripts/download_model.sh

We provide a small (200Mo) and a big (600Mo) version for each pretrained model (see training section for details). The command will download one of the following models:

Important notes :page_with_curl: 1. :exclamation:These small models correspond to an old version of the code, with in particular less training iterations. We release them for retrocompatibility and completeness, retrain from scratch for a thorough comparison. 2. it may happen that `gdown` hangs, if so you can download them manually with the gdrive links and move them to the `models` folder.

How to use :rocket:

1. Demo - 3D reconstruction of car images :oncoming_automobile:

example_car.png example_rec.gif

You first need to download the car model (see above), then launch:

cuda=gpu_id model=car_big.pkl input=demo ./scripts/reconstruct.sh

where gpu_id is a target cuda device id, car_big.pkl corresponds to a pretrained model, demo is a folder containing the target images. Reconstruction results (.obj + gif) will be saved in a folder demo_rec.

We also provide an interactive demo Open in Colab to reconstruct cars from single images.

2. Train models from scratch :runner:

shapenet.gif

To launch a training from scratch, run:

cuda=gpu_id config=filename.yml tag=run_tag ./scripts/pipeline.sh

where gpu_id is a device id, filename.yml is a config in configs folder, run_tag is a tag for the experiment.

Results are saved at runs/${DATASET}/${DATE}_${run_tag} where DATASET is the dataset name specified in filename.yml and DATE is the current date in mmdd format.

Configs and guidelines :high_brightness: Available configs are: - `sn/*.yml`, `sn_big/*.yml` for each ShapeNet category - `car.yml`, `car_big.yml` for CompCars dataset - `cub.yml`, `cub_big.yml` for CUB-200 dataset - `horse.yml`, `horse_big.yml` for LSUN Horse dataset - `moto.yml`, `horse_big.yml` for LSUN Motorbike dataset - `p3d_car.yml`, `p3d_car_big.yml` for Pascal3D+ Car dataset **:exclamation:NB: we advocate to always check the results after the first stage.** In particular for cases like birds or horses, learning can fall into bad minima with bad prototypical shapes. If so, relaunch with a different seed.
Small vs big model :muscle: We provide two configs to train a small and a big version of the model. Both versions give great results, the main benefit of the bigger model is slightly more detailed textures. The architecture differences are: - a shared backbone vs separate backbones - 32/128/128 vs 64/512/256 code sizes for shape/texture/background - 16 vs 64 minimal number of channels in the generators For faster experiments and prototyping, we advocate the training of the small version.
Computational cost :moneybag: On a single GPU, the approximate training times are: - roughly 3 days for ShapeNet on a V100 - roughly 10 days for real-image datasets on a 2080Ti

3. Reproduce our quantitative results :bar_chart:

A model is evaluated at the end of training. To evaluate a pretrained model (e.g. sn_big_airplane.pkl):

  1. move the model to a fake folder and rename it model.pkl (e.g. in runs/shapenet_nmr/airplane_big)
  2. point to the fake tag to resume from in the config (e.g. resume: airplane_big in airplane.yml)
  3. launch the training (and thus evaluation) with:
cuda=gpu_id config=sn_big/airplane.yml tag=airplane_big_eval ./scripts/pipeline.sh
Chamfer-L1 scores on ShapeNet :triangular_ruler: | airplane | bench | cabinet | car | chair | display | lamp | phone | rifle | sofa | speaker | table | vessel | mean | |---|---|---|---|---|---|---|---|---|---|---|---|---|---| | 0.110 | 0.159 | 0.137 | 0.168 | 0.253 | 0.220 | 0.523 | 0.127 | 0.097 | 0.192 | 0.224 | 0.243 | 0.155 | 0.201 |

For CUB, the built-in evaluation included in the training pipeline is Mask-IoU. To evaluate PCK, run:

cuda=gpu_id tag=run_tag ./scripts/kp_eval.sh

4. Train on a custom dataset :crystal_ball:

If you want to learn a model for a custom object category, here are the key things you need to do:

  1. put your images in a custom_name folder inside the datasets folder
  2. edit the config file custom.yml (or custom_big.yml) in the configs folder: this includes changing the dataset name to custom_name and setting all training milestones
  3. launch training with:
cuda=gpu_id config=custom.yml tag=custom_run_tag ./scripts/pipeline.sh

Further information :books:

If you like this project, check out related works from our group: