Question Regarding Color vs. Latent Image Fusion in TexFusion

Quantuman134 / TexPainter

TexPainter: Generative Mesh Texturing with Multi-view Consistency

MIT License

28 stars 2 forks source link

Question Regarding Color vs. Latent Image Fusion in TexFusion #4

Open costrice opened 2 weeks ago

costrice commented 2 weeks ago

Hi,

I've thoroughly read your paper and found it to be an excellent work with remarkable results. However, I have a question regarding a specific aspect of the method:

Does changing the latent image fusion in TexFusion to color image fusion in this work solve the challenges illustrated in Figure 2? In Figure 2, you highlight two challenges faced by TexFusion: 1. low-resolution latent texture produces blurry results, and 2. high-resolution latent texture produces inconsistent results as nearby views sample different texels. Wouldn't these challenges persist even after replacing the latent texture image with a color texture image?

I would greatly appreciate your clarification on this matter.

Thank you!

Quantuman134 commented 2 weeks ago

Thank you for your attention to our work! Yes, simply replacing the latent texture with color texture does not work. The point is fusing images of multiple views into the color texture. This operation ensures the consistency. We do not directly implement the fusion on latent texture because it is hard to find an idea way to fuse in latent space.

costrice commented 1 week ago

Thanks for your reply!

After reviewing TexFusion again, I noticed that it also aggregates the partially denoised latent image into a latent texture map after each denoising step, which is similar to "fusing images of multiple views into the color texture." This makes their fusion method quite similar to yours.

Given this similarity, I am still curious why their method encounters the problems shown in Figure 2, while yours does not?

Quantuman134 commented 4 days ago

A short explaination is, using a latent texture during generation is not a good idea. It causes the problem mentioned in Fig2. That is the reason we use a texture in color space. I think you could make a trial to encode a color texture to latent code, then render the latent texture and observe the rendered image decoded from latent code rendering. You would find the similar rendered result with the Fig2. Latent texture causes the blurry area.