black spots in the generated texture

[wangzhongliang]

first of all, it's a nice work definitely. I use it to generate different texture for car models it looks nice from far away

But in my case, there are some black spots in the generated texture as shown below I wonder how to fix it any help is appreciated, thx

here's my config:

mesh: "./car.glb"
mesh_config_relative: True    # mesh is placed relative to the config file
mesh_autouv: True
use_mesh_name: False
prompt: "A photo of a red race car."
negative_prompt: "shadow, light shadow, text"
steps: 30
cond_type: "depth"
mesh_scale: 1

in addition when i change cond_type to "normal" it gets better

[LIU-Yuxin]

A possible reason that leads to this problem is that the background colors are projected onto the object, we are using backpropagation feature of a differentiable renderers to propagate the colors from view space to texture space, which means a texel has to be visible in a view to have the color propagated. When one texel is visible in multiple view, we do a weighted combination based on viewing angle. You can imagine a neighbour pixels in view space could be disconnected in textures space, meaning that some pixels can be unfortunately, only visible in a bad angle (instead of receiving a color from a front-up view, it receives its color from a side view, making the texel prone to be polluted by background colors due to color bleeding in the decoded view). There should be better ways to tackle this problem, but we haven't tested those methods. For using normal conditioning, normally it will follow the shape of the mesh better, but seems that the training quality of normal controlnet is not as good as the depth one, so we didn't use it as default option.

[Yzhbuaa]

@wangzhongliang Hello, zhongliang, I also met this problem, have you found a solution? Thank you!

[Yzhbuaa]

@LIU-Yuxin The issue seems to stem from the weighted averaging in the UV space in the overlapping areas of multiple views. Is there a good solution to this problem?