Alexander W. Bergman,
Petr Kellnhofer,
Gordon Wetzstein,
Stanford University.
Official Implementation for Fast Training of Neural Lumigraph Representations using Meta Learning.
To get started, create a conda environment with all dependencies:
conda env create -f environment.yml
conda activate metanlrpp
The code is organized as follows:
Pre-train the encoder and decoder using the FlyingChairsV2 training dataset as follows:
python experiment_scripts/pretrain_features.py --experiment_name XXX --batch_size X --dataset_path /path/to/FlyingChairs2/train
Alternatively, use the checkpoint in the checkpoints directory.
Train a NLR++ model using the following command:
python experiment_scripts/train_sdf_ibr.py --config_filepath configs/nlrpp_dtu.txt --experiment_name XXX --dataset_path /path/to/dtu/scanXXX --checkpoint_img_encoder /path/to/pretrained/encdec
Note that we have uploaded our processed version of the DTU and NLR data here. The raw NLR data can be found here.
Meta-learn the initialization for the encoder, decoder, aggregation function, and neural SDF using the following command:
python experiment_scripts/train_sdf_ibr_meta.py --config_filepath configs/nlrpp_dtu_meta.txt --experiment_name XXX --dataset_path /path/to/dtu/meta/training --reference_view 24 --checkpoint_img_encoder /path/to/pretrained/encdec
Some optimized initializations for the DTU and NLR datasets can be found in the data directory. Additional models can be provided upon request.
Use the meta-learned initialization to specialize to a specific scene using the following command:
python experiment_scripts/test_sdf_ibr_meta.py --config_filepath configs/nlrpp_dtu_metaspec.txt --experiment_name XXX --dataset_path /path/to/dtu/scanXXX --reference_view 24 --meta_initialization /path/to/learned/meta/initialization
Test the converged scene on withheld views using the following command:
python experiment_scripts/test_sdf_ibr.py --config_filepath configs/nlrpp_dtu.txt --experiment_name XXX --dataset_path /path/to/dtu/scanXXX --checkpoint_path_test /path/to/checkpoint/to/evaluate
If you find our work useful in your research, please cite
@inproceedings{bergman2021metanlr,
author = {Bergman, Alexander W. and Kellnhofer, Petr and Wetzstein, Gordon},
title = {Fast Training of Neural Lumigraph Representations using Meta Learning},
booktitle = {NeurIPS},
year = {2021},
}
If you have any questions or would like access to specific ablations or baselines presented in the paper or supplement (the code presented here is only a subset based off of the source code used to generate the results), please feel free to contact the authors. Alex can be contacted via e-mail at awb@stanford.edu.