Modules, operators and utilities for 3D neural rendering in single-object, multi-object, categorical and large-scale scenes.
Pull requests and collaborations are warmly welcomed :hugs:! Please follow our code style if you want to make any contribution.
Feel free to open an issue or contact Jianfei Guo at ffventus@gmail.com if you have any questions or proposals.
An example of our platform (python=3.8, pytorch=1.11, cuda=11.3 / 11.7):
conda create -n nr3d python=3.8
conda activate nr3d
conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch
conda install pytorch-scatter -c pyg
pip install opencv-python-headless kornia imagesize omegaconf addict \
imageio imageio-ffmpeg scikit-image scikit-learn pyyaml pynvml psutil \
seaborn==0.12.0 trimesh plyfile ninja icecream tqdm plyfile tensorboard \
torchmetrics
cd
to the nr3d_lib
directory, and then: (Notice the trailing dot .
)
pip install -v .
:pushpin: NOTE: For pytorch>=2.2, c++17 standard is required --- in this case, you can run
USE_CPP17=1 pip install -v .
dLd[]
) and second-order gradients (d(dLdx)d[]
):rocket: All implemented with Pytorch-CUDA extension | dimension | forward | dL dparam |
dL dx |
d(dLdx) d(param) |
d(dLdx) d(dLdy) |
d(dLdx) dx |
---|---|---|---|---|---|---|---|
Dense |
2-4 | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
Hash hash-grids in NGP |
2-4 | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
VectorMatrix or VM Vector-Matrix in TensoRF |
3 | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
VecZMatXoY modified from TensoRF using only xoy mat and z vector. |
3 | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
CP CP in TensoRF |
2-4 | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
NPlaneSum "TriPlane" in EG3D |
3-4 | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
NPlaneMul |
3-4 | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
lod_res: [32, 64, 128, 256, 512, 1024, 2048, 4096]
lod_n_feats: [4, 4, 8, 4, 2, 16, 8, 4]
lod_types: [Dense, Dense, VM, VM, VM, CP, CP, CP]
lod_res: [[144, 56, 18], [199, 77, 25], [275, 107, 34], [380, 148, 47], [525, 204, 65], [726, 282, 91], [1004, 390, 126], [1387, 539, 174]]
lod_n_feats: [4, 4, 4, 4, 2, 2, 2, 2]
lod_types: [Dense, Dense, Hash, Hash, Hash, Hash, Hash, Hash]
log2_hashmap_size: 19
Check out docs/pack_ops.md for more!
Code: render/pack_ops
Code: render/raymarch/occgrid_raymarch.py
This part is primarily borrowed and modified from nerfacc
Code: models/attributes
We extend pytorch.Tensor
to represent common types of data involved in 3D neural rendering, e.g. transforms (SO3, SE3) and camera models (pinhole, OpenCV, fisheye), in order to eliminate concerns for tensor shapes, different variants and gradients and only expose common APIs regardless of the underlying implementation.
![]() |
![]() |
---|
These data types could have multiple variants but with the same way to use. For example, SE3 can be represented by RT matrices, 4x4 matrix, or exponential coordinates, and let alone the different representations of the underlying SO3 (quaternions, axis-angles, Euler angles...) when using RT as SE3. But when it comes to usage, the APIs are the same, e.g. transform()
, rotate()
, mat_3x4()
, mat_4x4()
, inv()
, default transform, etc. In addition, there could also be complex data prefix like [4,4]
or [B,4,4]
or [N,B,4,4]
etc. Once implemented under our framework and settings, you need only care about the APIs and can forget all the underlying calculations and tensor shape rearrangements.
You can check out models/attributes/transform.py
for better understanding. Another example is models/attributes/camera_param.py
.
Most of the basic pytorch.Tensor
operations are implemented for Attr
and AttrNested
, e.g. slicing (support arbitrary slice with :
and ...
), indexing, .to()
, .clone()
, .stack()
, .concat()
. Gradient flows and nn.Parameters()
, nn.Buffer()
are also kept / supported if needed.
fields
: single sceneCode: models/fields
sdf
neus
nerf
:
fields_conditional
: conditional / categorical / generative fieldsCode: models/fields_conditional
neus
:
fields_forest
: large-scale multi-continuous-block fieldsCode: models/fields_forest
neus
:
sdf
:
If you find this library useful, please cite our paper introducing pack_ops, cuboid hashgrids and efficient neus rendering.
@article{guo2023streetsurf,
title = {StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Views},
author = {Guo, Jianfei and Deng, Nianchen and Li, Xinyang and Bai, Yeqi and Shi, Botian and Wang, Chiyu and Ding, Chenjing and Wang, Dongliang and Li, Yikang},
journal = {arXiv preprint arXiv:2306.04988},
year = {2023}
}