Rasterizer implementation

[ioga]

Are there any plans to implement a rasterizer which can differentiably render a triangular mesh to an image, similar to https://github.com/google/tf_mesh_renderer or https://github.com/pmh47/dirt?

As far as I can see today, "Reflectance" and "Spherical Harmonics Approximation" colab demos implement what is basically a ray tracer, and the code is limited to computing intersection with only spheres and triangles. All visualizations use three.js renderer, which can't be used in differentiable manner in tensorflow.

One could implement a ray tracer for triangular meshes based on these utilities, but this seems terribly inefficient. So I wonder if this will be addressed in a later release, or is it out of scope for this project? Thank you!

[julienvalentin]

Hi,

We scheduled the release of a generic (python only) rasterizer, which you will be able to run on any compute architecture supported by TensorFlow. There is no precise ETA, but I would think that this should be available in approximately one month.

Depending on the run-time of the rasterizer described above and if there is enough interest, we will consider writing a GPU-only version.

I hope this answers your questions.

Best.

[julienvalentin]

I am closing this; feel free to re-open if you have further questions

[xuelin-chen]

Depending on the run-time of the rasterizer described above and if there is enough interest, we will consider writing a GPU-only version.

I hope this answers your questions.

Best.

Looking forward to that differentiable rasterizer, especially GPU version. :)

[frankhome61]

Hi! I am wondering if there is a new ETA of the differentiable rasterizer? I am working on a visual localization project and would be great to use the differentiable rasterizer!

[julienvalentin]

It is coded up and actually already pushed on GIT. It has been written in python / TensorFlow and hence is going to be available on any compute platforms supported by these (GPU / CPU etc.). Note nevertheless that the goal of this first version is to offer something generic so that everyone has access to the feature; if there is a lot of requests we will consider writing a fast GPU-only version.

Although it is deployed, it is not stable yet and still needs work. If you are happy working with experimental code, feel free to play with it and let us know if you experience any problems!

Best.

[iliaglazkov]

@julienvalentin Sounds awesome. Where can we find the experimental code? I don't see any relevant branches in this repo.

Thanks.

[frankhome61]

Also wondering if the documentation for relavant rasterization part is available? If so, where could I find them?

[julienvalentin]

The code + documentation currently lives on the main branch at https://github.com/tensorflow/graphics/blob/master/tensorflow_graphics/rendering/differentiable_renderer/rasterizer.py

[frankhome61]

Hi @julienvalentin, thanks for the update. I am currently working on a visual localization project and would like to know how I could utilize the current release of tensorflow graphics to render objects on screen? Thanks!

[francoisruty]

@julienvalentin thanks for the link, I'm going to test this on my use case and I'll get back to you if I find issues. I just have a small question: currently I use the old Google tf_mesh_renderer which does rasterization + phong shading. According to you, should I try to use TFG rasterizer and plug it to tf_mesh_renderer phong_shader, or should I try to do rasterization + shading in TFG?

So far I would tend to use TFG current rasterizer output and tf_mesh_renderer phong_shader, since TFG currently has on the rendering side a few utils function but nothing that takes a rasterizer output in input

[julienvalentin]

Hi Francois.

This ticket is closed; can I ask you to create a new one? That makes things easier for us to track.

Best.

[julienvalentin]

@frankhome61 , can I also ask you to create a new issue?

Best.

[francoisruty]

sure https://github.com/tensorflow/graphics/issues/28