search

SpicyRamenDev / diffusion-nerf

An implementation using NVIDIA Kaolin Wisp and Stable Diffusion that applies the Score Distillation Sampling to the task of Few-shot / single-view 3D object reconstruction.
37 stars 3 forks source link

readme

Few shot 3D reconstruction using Score Distillation Sampling

An implementation using NVIDIA Kaolin Wisp and Stable Diffusion that applies the Text-to-3D model DreamFusion to the task of Few-shot / single-view 3D object reconstruction.

Examples

Few-shot 3D object reconstruction

Input RGB views (only 3 sparse views, omitting the right side of the car): 1 2 3
car_rgba_00002 car_rgba_00000 car_rgba_00001

L2 reconstruction loss on the input views + Score Distillation Sampling with the guiding prompt "a yellow sports car":

car_reconstruction_1.webm

Text-to-3D

Score Distillation Sampling with a guiding prompt:

1 2 3
croppedsample_3_angle_oh croppedsample_3 croppedsample_3_normal
croppedsample_2 croppedsample_2_normal croppedsample_2_albedo
croppedsample_1 croppedsample_1_normal croppedsample_4

Installation

Follow the instrutions here to install NVIDIA Kaolin Wisp.

Run setup.sh to install the prerequisites.

Usage

Few-shot 3D object reconstruction

python main.py --config configs/diffusion_nerf.yaml --dataset-path /path/to/car/dataset --prompt "a car"

Text-to-3D

python main.py --config configs/diffusion_nerf.yaml --prompt "a DSLR photo of a blue car"