search

ashawkey / stable-dreamfusion

Text-to-3D & Image-to-3D & Mesh Exportation with NeRF + Diffusion.
Apache License 2.0
8.05k stars 713 forks source link

dmtet finetuning results in round density maps #218

Open anime26398 opened 1 year ago

anime26398 commented 1 year ago

Description

After training using zero123 with catstatue image, when trying to fine-tune using dmtet, the model produces round rgb and depth predictions, even after complete finetuning

Steps to Reproduce

After training using python main.py -O --image ./data/catstatue_rgba.png --workspace cat_toy_trial --iters 5000

When trying to fine-tune using dmtet python main.py -O --image ./data/catstatue_rgba.png --workspace cat_toy_trial_dmtet --dmtet --init_ckpt cat_toy_trial/checkpoints/df_ep0050.pth it produces round rgb and depth predictions

df_ep0039_0005_depth df_ep0039_0005_rgb

Expected Behavior

unexpected

Environment

Ubuntu 22.04, torch 1.12.1, CUDA 11.6

ashawkey commented 1 year ago

@anime26398 Hi, this seems to be a bug with torch2, you can refer to https://github.com/ashawkey/stable-dreamfusion/issues/217 for a temporary solution. I'll fix this soon.

anime26398 commented 1 year ago

I am using torch 1.12.1 though (instead of torch2).

Also, you had mentioned in that issue that "you can fix it by deleting the torch.compile block in utils.py". Which utils.py file are you mentioning here, I could not find it in the repo.

Thanks!

anime26398 commented 1 year ago

got it now i have commented these lines in nerf/utils.py

# try out torch 2.0
# if torch.__version__[0] == '2':
#     self.model = torch.compile(self.model)
#     self.guidance = torch.compile(self.guidance)
ashawkey commented 1 year ago

If you are using torch 1.12, this is not likely the solution. Could you check if it becomes a sphere at the first epoch (in validation folder), or during training?

anime26398 commented 1 year ago

Somehow commenting out these lines solved the issue. Now it is working as expected. Thanks!