search

threestudio-project / threestudio

A unified framework for 3D content generation.
Apache License 2.0
6.19k stars 474 forks source link

Proper score debiasing value #101

Open SusungHong opened 1 year ago

SusungHong commented 1 year ago

Great project! I had a lot of fun experimenting with it and am excited about the score debiasing implementation. By testing the dynamic scale ranging from 0.5 to 2 (considering the lower scale of the gradient), I noticed a significant reduction in artifacts. Some examples from this are included in the project page.

Although it may lead to slightly longer steps for someobjects, I believe it would be highly beneficial to include this technique in the 'Tips' section of the README. Also, it would be great to provide a link to our paper for further reference.

bennyguo commented 1 year ago

Thank you @SusungHong !

About the dynamic range, I've tried 2 to 8 as suggested in the paper and it indeed helps in many cases. Will 0.5 to 2 give better results? Also, did you use a smaller learning rate for the network? —— because in the stable-dreamfusion repo they adopt a learning rate of 0.01 for the encoding and 0.001 for the network. This slows down convergence but could also alleviate the multi-face problem in some cases.

I'd love to include this technique in the tips and give a link to your paper. Would you be interested in opening a PR about this?

BTW, we also have a PR implementing prompt debiasing :)

SusungHong commented 1 year ago

I haven't measured it quantitatively, but 0.5 to 2 provides us with noticeable examples for this implementation. This is also reasonable because the scale of the score is usually less than a quarter of that of the SJC repo. 2~8 is also good, though.

I've also observed that the learning rate affects the Janus problem. It could be an option, but score debiasing has the advantage of requiring a similar number of iterations, with a few exceptions (a flamingo's neck in my case; it grows slowly).

I've opened a pull request here #102. Great that you have already been implementing the prompt debiasing!

bennyguo commented 1 year ago

Great! I've merged your PR. And I think I could do some experiments on the 0.5-2 guidance scale. If it brings consistent improvements for SDS, I'll consider making it the default :) I'll just leave this issue open.