search

threestudio-project / threestudio

A unified framework for 3D content generation.
Apache License 2.0
6.32k stars 480 forks source link

batch_size cannot be larger than 1 in controlnet guidance #250

Open dunbar12138 opened 1 year ago

dunbar12138 commented 1 year ago

https://github.com/threestudio-project/threestudio/blob/94c6ad2a4e391e8a85ac28724870bcd515b9e591/threestudio/models/guidance/controlnet_guidance.py#L347

Hi, thanks for your great work. Just have a quick question: is there a specific reason that batch_size here cannot be larger than 1?

DSaurus commented 1 year ago

Hi, @dunbar12138. ControlNet support both SDS and DU(dataset update proposed by instructnerf2nerf). In the DU setting, ControlNet needs to define a scheduler with a random timestep to execute multiple sampling steps for obtaining an RGB image. If the batch size is not 1, the randomly generated tilmestep may differ within the same batch, which is not permissible for a single scheduler. However, if you are only utilizing the SDS loss, I believe you can comment out this code.

dunbar12138 commented 1 year ago

Thanks for your reply! I'll test SDS loss with larger batch_size then