This repository contains a pytorch implementation for the paper: TensoIR: Tensorial Inverse Rendering.
The code can run well, but it is not well organized. I may re-organize the code when I am available.
Install environment:
conda create -n TensoIR python=3.8
conda activate TensoIR
pip install torch==1.10 torchvision
pip install tqdm scikit-image opencv-python configargparse lpips imageio-ffmpeg kornia lpips tensorboard loguru plyfile
Please download the dataset and environment maps from the following links and put them in the ./data
folder:
We provide the code for generating your own synthetic dataset with your own blender files and Blender software. Please download this file and follow the readme.md file inside it to render your own dataset. The Blender rendering scripts heavily rely on the code provided by NeRFactor. Thanks for its great work!
normals_diff_weight
, which controls the how close the predicted normals should be to the derived normals. Larger weight will help preventing the normals prediction from overfiting the surpervised colors, but at the same time it will demage the normals prediction network's ability to predict high-frequency details. We recommend three values to try: 0.0005
, 0.002
, and 0.005
when you train TensoIR on your own dataset.export PYTHONPATH=. && python train_tensoIR.py --config ./configs/single_light/armadillo.txt
export PYTHONPATH=. && python train_tensoIR_rotated_multi_lights.py --config ./configs/multi_light_rotated/hotdog.txt
export PYTHONPATH=. && python train_tensoIR_general_multi_lights.py --config ./configs/multi_light_general/ficus.txt
We don't do quantitative and qualitative comparisons for the original NeRF-Synthetic dataset in our paper (the reasons have been discussed above), but you can still train TensoIR on the original NeRF-Synthetic dataset for some analysis.
export PYTHONPATH=. && python train_tensoIR_simple.py --config ./configs/single_light/blender.txt
export PYTHONPATH=. && python "$training_file" --config "$config_path" --ckpt "$ckpt_path" --render_only 1 --render_test 1
"$training_file"
is the training script you used for training, e.g. train_tensoIR.py
or train_tensoIR_rotated_multi_lights.py
or train_tensoIR_general_multi_lights.py
.
"$config_path"
is the path to the config file you used for training, e.g. ./configs/single_light/armadillo.txt
or ./configs/multi_light_rotated/hotdog.txt
or ./configs/multi_light_general/ficus.txt
.
"$ckpt_path"
is the path to the checkpoint you want to test.
The result will be stored in --basedir
defined in the config file.
export PYTHONPATH=. && python scripts/relight_importance.py --ckpt "$ckpt_path" --config configs/relighting_test/"$scene".txt --batch_size 800
We do light intensity importance sampling for relighting. The sampling results are stored in --geo_buffer_path
defined in the config file.
"$ckpt_path"
is the path to the checkpoint you want to test.
"$scene"
is the name of the scene you want to relight, e.g. armadillo
or ficus
or hotdog
or lego
.
Reduce the --batch_size
if you have limited GPU memory.
The line 370 of scripts/relight_importance.py
specifies the names of environment maps for relighting. You can change it if you want to test other unseen lighting conditions.
The mesh will be stored in the same folder as the checkpoint.
export PYTHONPATH=. && python scripts/export_mesh.py --ckpt "$ckpt_path"
If you find our code or paper helps, please consider citing:
@inproceedings{Jin2023TensoIR,
title={TensoIR: Tensorial Inverse Rendering},
author={Jin, Haian and Liu, Isabella and Xu, Peijia and Zhang, Xiaoshuai and Han, Songfang and Bi, Sai and Zhou, Xiaowei and Xu, Zexiang and Su, Hao},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2023}
}
The code was built on TensoRF. Thanks for this great project!