how to render complex multiple objects scene?

[yuedajiong]

hi, developers: I have referred nvdiffmodeling and nvdiffrecon， implemented the vertex-color rendering, and fixed uv-textures rendering issues(such as no usemlt *** in .obj), that is, now, i can render vertex-color and uv-texture, but how can i render multiple objects in one same scene, that means, I need the depth information from camera view. where i can get the depth information?
dr.rasterize or dr.DepthPeeler().rasterize_next_layer #(u, v, z/w, triangle_id) normalized depth value z/w is used later by the antialiasing operation to infer occlusion relations between triangles

I need you guys help, and I can contribute the final model back to you: nvdiffrast based multi-format(vertex-color and uv-texture hybird) multi-object (fore-back, inside-outside(body and clouth)) complex rendering.

thanks and hope you help: where i can get the depth information?

[yuedajiong]

I have implemented the super differentable-render based on nvdiffrast: different-format(vertex-color, uv-texture, ...), multi-object (fore-back, in-out/contain, ...), support batch-operation, including light, ...

anyway, thanks Nvidia.

[s-laine]

The main complication is that the antialias op should be run on the final image, but it isn't designed to understand images that are composited from multiple rendering passes.

I'm assuming you don't have any transparency in the scene. In this case, you can composite multiple color buffers by choosing the pixel value from the pass that has the lowest z/w in rasterization and contains a surface — this will give you the color of the closest surface in that pixel. For the antialias op to work properly for the composite image, you need to do a similar composition step for the raw rasterizer outputs, and feed that to the antialias op.

However, the antialias op takes just one vertex buffer and one triangle buffer. If all rasterization is done from common vertex/triangle buffers, just using different triangle ranges, everything should work automatically. If not, you will have to combine the vertex/triangle buffers used in different rendering passes into one consolidated buffer for vertices and one for triangles. Note that you'll have to adjust the vertex IDs in the triangle buffers and the triangle IDs of the rasterizer output buffers according to the offset they will have in the final, consolidated buffers, so that all indices end up pointing to the right place.

[yuedajiong]

thanks s-laine.
2023-04-25, i have completed the argmin(... z/w), and picked colors from multiple objs color channels, no transparency requirements so far. it looks it is ok.

but i still need to process the alpha component in rgba. and the later antialias-op.

batch = []
for b in range(rendered_rasterize_all.shape[0]):
    rendered_rasterize_batch_one = rendered_rasterize_all[b]
    rendered_rasterize_batch_one_depth = rendered_rasterize_batch_one[:,:,:,2]
    min_index = torch.argmin(rendered_rasterize_batch_one_depth, dim=0, keepdim=False).unsqueeze(2)
    min_index = min_index.repeat(1,1,4).unsqueeze(0)
    rendered_layer_batch_one = rendered_layer_all[b][:,:,:,0:4]
    min_layer = torch.gather(rendered_layer_batch_one, dim=0, index=min_index)[0]
    min_layer = min_layer[:,:,0:4]  #*255.
    batch.append(min_layer)
batch = torch.stack(batch, dim=0)

vertex-3.obj:

v -0.500000 +0.000000 -0.500000 255 0 0 v +0.500000 +0.000000 -0.500000 0 255 0 v +0.000000 +0.500000 -0.500000 0 0 255 v +0.000000 +0.000000 +0.000000 255 255 0

f 1 2 3 f 1 2 4 f 1 3 4 f 2 3 4

vertex-4.obj: v -1.000000 -0.100000 -1.100000 64 96 96 v +1.000000 -0.100000 -1.100000 96 64 96 v +0.000000 +0.900000 -1.100000 96 96 64 v +0.000000 -0.100000 -0.100000 64 64 96 v +0.000000 -1.100000 -1.100000 96 64 64

f 1 2 3 f 1 2 4 f 1 3 4 f 2 3 4 f 1 2 5 f 1 4 5 f 2 4 5

and another vertex-4-uv.obj.

[yuedajiong]

hi, s-laine: I have completed the super-render based on you guys nvdiffrast: multi-format, multi-object. here is my output image: vertex-3 fore and embed-in vertex-4. rendered ok. I will try to understand you mentioned that 'antialias'-op. in my implementation, i have done that op, maybe I missed something. very happy. if I can submit my code, I wil be one fo contributors of the library. thank s-laine.

[yuedajiong]

hi, s-laine: I rendered my hybird-format multi-object as follow, still have a quesion:
I rendered my objs one by one, including composite_buffer (including antialias), and pick and merge the rendered image by closest information in 'rasterize'.
But, the final output image has zigzag edge. my question is, do 'antialias' again before return? thanks.

multi-object: vertex-color{body, hair}, uv-texutre{garment-under, garment-upper}

[s-laine]

The antialiasing operation should be executed only on the final, composited image, as explained in my previous message. Otherwise you won't get correct silhouette gradients where one object is behind another.

We're not accepting external contributions to the library.

[yuedajiong]

got it. thanks s-laine again.

[csyhping]

Hi @s-laine , if I just concatenate several meshes into one (something like trimesh.concatenate， and use the combined mesh's information for rasterization, can I get a correct rendering?

I tried with three meshes: A and B are foreground meshes, and C is background meshes. It is pretty weird, and the order of concatenate operation matters. If I concatenate [A, C, B], then the B is not seen in the rendered image; if I concatenate [B, C, A], then A is not. If [C, B, A], then both AB are not seen. Do you have any ideas about this? Thanks.

[s-laine]

Answered here.