[Question] How to project a 2D instance Segmentation mask on a 3D Model

[AliButtar]

Describe the issue

I have a 3D reconstruction of an object which also has some extra view than required.

Here is my supposed pipeline:

1. Given a 3D Reconstruction. I want to place cameras around the object and then take images of the object.
2. Use those images with Gounding SAM to get a black-and-white instance segmentation mask of the object of interest.
3. Use the pixels of the mask and project it back on the 3D Reconstruction ( (x, y) would become (x, y, z) )
4. Use the mask in the 3D Reconstruction to get the measurement of the object from the side where the shot was taken

In simple words, it is just mapping (X, Y) of a 2D image to 3D and getting the (X, Y, Z). Now I know step 1 can be done using Blenderproc. What I want to know is if Blenderproc can help with the rest of the steps, especially the projection part.

Looking forward to the response.

Thanks

Minimal code example

No response

Files required to run the code

No response

Expected behavior

The mask accurately gets projected on the 3D reconstruction and perhaps can be visualized

BlenderProc version

latest

[cornerfarmer]

Hey @AliButtar,

in #1045 I added a new function bproc.camera.unproject_points() which might be what you need.