Edge bug for optimizing texture with vertex color / non interpolated vertex color

[dawnKalo]

Hi,

I try to optimize texture with vertex color for the visible part of the mesh. I define the camera parameters and optimize ONLY for the texture with render_albedo, I just want to fill the mesh with the texture of the image excluding the invisible part. I got very good result, excluding the edges:

I want the edge will be sharp: dark for the invisible part and the correct texture of the image for the visible part, without the stripe between them.

I've used 1 sample per pixel but I thought that the problem occurs because the renderer samples different ray for each iteration, so I set "sample_pixel_center" to True in order to get the same vertex in each iteration, but it didn't solve the problem.

Second thought about it is that perhaps the vertex color interpolation is the problem. It leads me to the alternative way to solve it: is there a way to get directly the closest vertex index of the object? I thought to set texture as the vertex index, but the interpolation changes the value and there is no meaning to the continuous value of vertex indices (unlike position which continuous value is the correct approach). I guess it's not very difficult modification, I attach here the suggested code, but I guess there is a need to change also the differentiable part.

// Interpolate color
auto cc = Vector3{0, 0, 0};
if (has_colors(shape)) {
   auto c0 = get_shading_normal(shape, ind[0]);
   auto c1 = get_shading_normal(shape, ind[1]);
   auto c2 = get_shading_normal(shape, ind[2]);
   auto maxV = max(max(w , u) , v)
    if (!has_colors_interpolation(shape)) {
        switch (maxV)
        {
            case w: 
                w = 1;
                v = 0;
                u = 0;
                break;
            case v:
                w = 0;
                v = 1;
                u = 0;
                break;
            case u:
                w = 0;
                v = 0;
                u = 1;
                break;
        }
    }
    cc = w * c0 + u * c1 + v * c2;
}

[BachiLi]

What you proposed is not differentiable due to the conditional switch statement. One way to do this in a differentiable way is to take the weights to the power of p and normalize (cc = (w^p c0 + u^p c1 + v^p c2) / (w^p + u^p + v^p)). If you want to implement this go ahead, otherwise I can do it after 3~4 days from now.

[dawnKalo]

Thanks for the response. The switch case is not differentiable but it's true only when we optimize wrt to weights. On the contrary, it is differentiable with respect to vertex color even for the switch case. There is very common area in computer graphics that find correspondence between vertex index to semantic area, so the goal is optimizing such that for example vertex number Vi will be in position of (xi,yi,zi). For that I need to get the closest vertex index for each pixel, if I will not use the switch case, so let's assume I have triangle face with vertices of (0,1,10000), so even if p is large ~0.999, so I'll get 9990 which is different vertex (in addition, it's not longer int type). I mean for that part I don't need the differentiability, I just need to find the vertex index and then optimize for that: Vi = vertices_rendered[landmarkY,landmarkX].cpu().detach().numpy() loss = (vertices[ Vi , :] - desiredPos).pow(2).mean() Not that the optimization is wrt the position of vertex vi but not optimizing on the vertex index.

In summary, I suggest to give the user the opportunity to choose p, in case he chooses without interpolation (lets indicate it by -1), so we will use switch case, but he cannot optimize it with respect to weights (only for vertex color) and if he'll try, he'll get assertion.

Regarding the code, I attach here the suggested code, I guess there is more modifications in order to make it work, so I'll wait for you to complete it and I learned from your modifications for other changes I would like to change in the future.

p.s I'm going to use Redner in the near future and I guess I'll have more ideas to improve it. However, this is the first time I'm dealing with interface between python and c++ and I know how to debug python solely (with PyCharm) and c++ solely but I don't know how to debug it together, so if you can, please recommend me about debug interface. I have to say that I'm not fully understand how the python and C++ works together, for example I've installed the Redner with pip and I find the python files in my virenv/lib/python folder but I don't find the C++ files and in case I'll find them, so I need also to compile them and adapt them to the python. If you can explain me how it works, show example or short tutorial or refer me where to read about it so it would be helpful and after I'll understand it deeply, I would be able to contribute more easily to that project so it would worth it.

// Interpolate color
auto cc = Vector3{0, 0, 0};
if (has_colors(shape)) {
   auto c0 = get_shading_normal(shape, ind[0]);
   auto c1 = get_shading_normal(shape, ind[1]);
   auto c2 = get_shading_normal(shape, ind[2]);

    if (has_colors_interpolation_p(shape) {
            // p interpolation (differentiable)
            if (p_exp(shape) > 0) {
                w = pow(w, p_exp(shape));
                u = pow(u, p_exp(shape));
                v = pow(v, p_exp(shape))
                auto denominator = w + u + z;
                w /= denominator
                u /= denominator
                v /= denominator

            // nearest vertex weight (non differentiable)
            if (p_exp(shape) == -1) {
               auto maxV = max(max(w , u) , v)
               switch (maxV)
                {
                    case w:
                        w = 1;
                        v = 0;
                        u = 0;
                        break;
                    case v:
                        w = 0;
                        v = 1;
                        u = 0;
                        break;
                    case u:
                        w = 0;
                        v = 0;
                        u = 1;
                        break;
                }
            }
    cc = w * c0 + u * c1 + v * c2;
}

[BachiLi]

I added a feature to render barycentric coordinates in 0.4.15. If you specify barycentric_coordinates as an output rendering channel, 2 of the channels correspond to the u, v in your code. You can also output shape_id and triangle_id for each pixel. This allows you to implement what you proposed in pytorch. Let me know if you have trouble implementing this.

[dawnKalo]

Great, exactly what I need. I'll check it, thanks