Explore label rendering with `TexturedPhotogrammetryMesh.render_pyvista()`

[russelldj]

The render_pyvista method allows you to generate realistically-styled images from a camera viewpoint. We currently only use it to generate simulated images in the concept figures notebook and not in the core workflow. We should consider whether it can be used in the label rendering stage of the main workflow.

Since pytorch_3d uses a differentiable renderer, it may have additional overhead compared to a traditional rendering pipeline. We use pytorch_3d because this differentiable rendering exposes the pixel-to-face correspondences we need for texture aggregation. But in the label rendering stage, we don't need this information and another rendering pipeline might be faster.

To make render_pyvista work, we'd have to make sure that it can render integer labels correctly. Some things to look into would be

• [ ] Can "flat" lighting be turned on so there is no impact on the labels?
• [ ] Can interpolation between two adjacent values be turned off, so we don't get spurious intermediate values?
• [ ] Is this pipeline actually faster than using pytorch_3d? Because pytorch3d is GPU enabled, I'm not sure.

There may be other considerations as well

[Ruprabhu25]

By turning off diffuse lighting (setting default 1.0 to 0.0) and enabling ambient lighting (setting default 0.0 to 0.5), we get something like this. These parameters are set in plotter.add_mesh(). I still need to check how it compares against pytorch3d, but seems promising.

[Ruprabhu25]

I noticed that slight shadows were being cast on the ground mesh / on other shapes, which I found to be caused by ssao. Setting enable_ssao parameter to false in the render_pyvista fixed this.

[russelldj]

This seems really promising. In my mind, the next step would be figuring out the best way to avoid colormapping values. I'm pretty sure duplicating the texture as three channels would work. But maybe there's an easier way with what you know about explicitly controlling the colormap.

[russelldj]

We got to the point where we could do flat label rendering, but to do so we had to duplicate the one-channel label into three channels. We should benchmark whether there are major memory or performance penalties to this, or if there's another option.