search

SpicyRamenDev / diffusion-nerf

An implementation using NVIDIA Kaolin Wisp and Stable Diffusion that applies the Score Distillation Sampling to the task of Few-shot / single-view 3D object reconstruction.
34 stars 2 forks source link

Paper of your work #1

Open TontonTremblay opened 1 year ago

TontonTremblay commented 1 year ago

Hey very cool work, I was wondering if you could describe further the method you developed. Is it a 1-1 dream fusion or you did some other things? Again thank you for the great work.

timsterzizzle commented 1 year ago

@TontonTremblay how did you get it up and running?? what config are you using?

TontonTremblay commented 1 year ago

I did not try yet. I will try in the next couple weeks.

On Thu, Feb 9, 2023 at 17:46 timsterzizzle @.***> wrote:

@TontonTremblay https://github.com/TontonTremblay how did you get it up and running?? what config are you using?

— Reply to this email directly, view it on GitHub https://github.com/SpicyRamenDev/diffusion-nerf/issues/1#issuecomment-1425063690, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABK6JIFZBQTCLTAANDCRYXDWWWMWFANCNFSM6AAAAAAUWBM4JE . You are receiving this because you were mentioned.Message ID: @.***>

SpicyRamenDev commented 1 year ago

Thank you very much for your interest, and I apologize for the late reply. This repo is part of an ongoing graduate internship, so there is no paper attached to it, and the code and the results themselves are not really exploitable yet. I tried to follow closely the ideas presented in DreamFusion, with only minor differences in implementation (and no learnable background MLP yet at the moment). For the few-shot reconstruction task, in the repo there is only a photometric L2 loss for the input views on top of the SDS loss on novel views, but I am still trying to figure out better regularizations / constraints.

If you are having trouble running it, I ran the version in this repo on a docker image built with nvcr.io/nvidia/pytorch:22.10-py3, kaolin 0.12.0 and kaolin-wisp commit 351a04. Please note that the examples I provided were generated during development and with various configs, so you may need to tweak the parameters to obtain acceptable results.

kxhit commented 1 year ago

@SpicyRamenDev Hi, great work and open-sourced code! It's a very clean base code for other people to work on it! Thanks!

@timsterzizzle I managed to make it run on the newest wisp and diffusers>=0.10.0 by updating some parts of this repo's code. For example, change self.iteration == 1: -> self.iteration == 0:, data['imgs'] -> data['rgb'], MultiviewDataset -> NeRFSyntheticDataset. Hope it helps.