Official PyTorch Implementation of paper "Vision Transformer for NeRF-Based View Synthesis from a Single Input Image", WACV 2023.
Kai-En Lin1* Lin Yen-Chen2 Wei-Sheng Lai3 Tsung-Yi Lin4** Yi-Chang Shih3 Ravi Ramamoorthi1
1University of California, San Diego, 2Massachusetts Institute of Technology, 3Google, 4NVIDIA
* Work done while interning at Google, ** Work done while at Google.
Project Page | Paper | Pretrained models
Make sure you have up-to-date NVIDIA drivers supporting CUDA 11.1 (10.2 could work but need to change cudatoolkit
package accordingly)
Run
conda env create -f environment.yml
conda activate visionnerf
Clone the repository git clone https://github.com/ken2576/vision-nerf.git
and download dataset from PixelNeRF.
Download pretrained model weights.
Here is a list of the model weights:
nmr_500000.pth
: Our pretrained weights for the category-agnostic experiment.srn_cars_500000.pth
: Our pretrained weights for the category-specific experiment on ShapeNet Cars.srn_chairs_500000.pth
: Our pretrained weights for the category-specific experiment on ShapeNet Chairs.Install requirements conda env create -f environment.yml
.
Setup configurations in configs
.
(Optional) Run training script with python train.py --config [config_path]
.
The code also supports DDP and it can be run by
python -m torch.distributed.launch --nproc_per_node=[#GPUs] train.py --config [config_path] --distributed
Run inference script with our pretrained models:
python eval.py --config [path to config file] # For ShapeNet Cars/Chairs
python eval_nmr.py --config [path to config file] # For NMR
python gen_real.py --config [path to config file] # For real car data
Our pretrained model works with real car images. You can prepare the data using the same process as PixelNeRF.
Then, run gen_real.py
similar to the above example.
This code is based on DPT, IBRNet and PixelNeRF.
@inproceedings {lin2023visionnerf,
booktitle = {WACV},
title = {Vision Transformer for NeRF-Based View Synthesis from a Single Input Image},
author = {Lin, Kai-En and Yen-Chen, Lin and Lai, Wei-Sheng and Lin, Tsung-Yi and Shih, Yi-Chang and Ramamoorthi, Ravi},
year = {2023},
}