LIU-Yuxin / SyncMVD

Official PyTorch & Diffusers implementation of "Text-Guided Texturing by Synchronized Multi-View Diffusion"
MIT License
104 stars 5 forks source link

Latent Texture from middle steps #15

Open Jarvisss opened 3 weeks ago

Jarvisss commented 3 weeks ago

Hi, here is a question, I want to bake latent texture from middle steps. For example, run 10 steps without UV projection, and start the UV projection from step 11.

The following code you provide seems lose the noise distribution in the original latent, due to the sparsity of gradient and the voronoi solve operation. https://github.com/LIU-Yuxin/SyncMVD/blob/8750811f571276a4c5d137c2cfd7c2e39d1ba62d/src/syncmvd/step.py#L75C1-L78C62

if texture is None:
    sample_views = [view for view in sample]
    sample_views, texture, _ = uvp.bake_texture(views=sample_views, main_views=main_views, exp=exp)
    sample_views = torch.stack(sample_views, axis=0)[:,:-1,...]

Can I use the baked original texture and add noise to get the noisy texture instead? $x_t=\sqrt{\alpha_t}x_0+\sqrt{1-\alpha_t}\epsilon$

Any advice can be helpful, thanks!

LIU-Yuxin commented 1 week ago

Yes, you can output the original latent views and use it for baking, then add noise. The bake function is not intended to be used for noisy latents.