Does the system have any support for refining textured objects?

[reynoldscem]

Hi,

I was wondering if this was something you'd tried. E.g. initialising with some low-quality texture map and refining it with your system? Or taking some known camera poses with associated views into account?

If neither of these, would you foresee any big obstacles that might prevent someone implementing this? If not, I'll give one or both a go.

[LIU-Yuxin]

It should be possible to do both, but not sure how well it will work. For refinement, perhaps you need to render the objects from the predefined views and encode with the vae encoder first, then project the latents to the texture space. After that, apply noise to timestep t and refine using the current pipeline to generate a t0 noiseless texture. For known camera poses, perhaps you need to add that camera to the camera list, and modify the code to fix the content in that camera (like in an 2D inpainting method), and generate the rest of the views?

[reynoldscem]

Thank you for the reply and for the helpful suggestions!

[Yzhbuaa]

Hi @reynoldscem, I am curious to know if you have tried refining the coarse texture. If yes, could you share some insights about it?

[Yzhbuaa]

@LIU-Yuxin Could you explain how to 'apply noise to timestep t' after 'project the latents to the texture space'? Thank you.

[LIU-Yuxin]

@LIU-Yuxin Could you explain how to 'apply noise to timestep t' after 'project the latents to the texture space'? Thank you.

Since the latent texture has four channels, modifying the time step of the texture would be similar to modifying a latent image. You can either use the 'add_noise' method of the scheduler to set time step from 0 to any t, or implement a function following the mathematics of the forward diffusion process to set time step from any intermediate t to t'.

[reynoldscem]

@Yzhbuaa I haven't I'm afraid! But I still might get round to it at some point