use_geo = -1
in config files..txt
form, which can be enabled with use_geo = 1
and pointfile = /your/file.txt
in config files. preprosessing
folder, which is initially for our ealy trying and not used here../configs/synthetic-nerf/default/chair.txt
for the usage of hyperparameters. For Synthetic-NeRF dataset, we provide the initial geometry from DVGO (which is the default one in our implementation) and from MVS. Feel free to try both (e.g., use_geo = 1
and pointfile = /your/mvs_file.txt
) to see the comparison.
For Scannet dataset, we use the initial geometry provided by the dataset itself. We convert the original .ply
file into .txt
and you may download from here.
All the codes are tested in the following environment:
And the layout should look like this:
Strivec
├── data
│ ├── nerf_synthetic
│ │ |──default
│ │ │ |──chair
│ │ │ │──drums
│ │ │ |──...
│ │ |──local_vm
│ │ │ |──chair
│ │ │ │──drums
│ │ │ |──...
├── scene0101_04 (scannet)
│ │ │──exported
│ │ │──scene0101_04_2d-instance-filt.zip
│ │ │──...
├── scene0241_01 (scannet)
│ │ │──exported
│ │ │──scene0241_01_2d-instance-filt.zip
│ │ │──...
├── TanksAndTemple
│ │ │──Barn
│ │ │──Caterpillar
│ │ │──...
├── 360 (Mip-NeRF360)
│ │ │──garden
│ │ │──room
│ │ │──...
We not only provide the training and evaluation code to reproduce the results in the paper, but also the code of ablation that uses local VM tensors instead of local CP tensors (results are here).
# hierachical Strivec, without rotation (grid aligned)
python train_hier.py --config ./configs/synthetic-nerf/default/chair.txt
# local VM tensors instead of local CP tensors
train_dbasis.py --config ./configs/synthetic-nerf/local_vm/chair.txt
We visualize the local tensors of different scales into ./log/your_scene/rot_tensoRF/0_lvl_k.ply
, where k is the kth scale.
Here is a toy example to illustrate the TensoRF-CP (TensoRF-VM is similar) with global decomposition in (left) axis-aligned and (right) non-axis-aligned situations. The bottom shows the grid values. In axis-aligned case, only 1 component is needed to represent the scene (vector bases recover grid values by outer product). In non-axis-aligned case, however, 3 components are needed because the rank of matrix changes from 1 to 3 after scene rotation. While our design with local low-rank tensors can alleviate this issue, i.e., local tensors (2*2) are always rank-1 before and after rotation.
If you find our code or paper helps, please consider citing:
@INPROCEEDINGS{gao2023iCCV,
author = {Quankai Gao and Qiangeng Xu and Hao Su and Ulrich Neumann and Zexiang Xu},
title = {Strivec: Sparse Tri-Vector Radiance Fields},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
year = {2023}
}