How to render the triangle mesh to depthmap in different views?

[XYZ-qiyh]

Hi @JiayuYANG , thanks to your amazing work! I'm wondering how to render the mesh to the depthmap given the camera params? Which software or tool do you use? Looking forward to your reply!

[JiayuYANG]

Hi @XYZ-qiyh

I found pyrender quite useful. Open3D might also be good, I haven't tried it yet.

I render mesh to depth maps by:

• Import mesh into pyrender
• create virtual cameras from the camera parameters provided by DTU
• render the depth map by pyrender.OffscreenRenderer()

You can check the documentation of pyrender for details.

I'll also release code for this purpose. But I'm not sure when can I finish cleaning and organizing the code. You can make your own implementation if you are in hurry. You can start from here.

Jiayu.

[XYZ-qiyh]

Thanks for your prompt reply and detailed guidance, I'll try it later.

[XYZ-qiyh]

Hi, @JiayuYANG , sorry to bother you again. I have some problems with the underlying engine of the pyrender. When I run the offscreen rendering demo using EGL, it raises an error like this :

  File "offscreen_render.py", line 26, in <module>
    r = pyrender.OffscreenRenderer(400, 400)
  File "/mnt/A/qiyh/2022/Pyrender/pyrender-master/pyrender/offscreen.py", line 31, in __init__
    self._create()
  File "/mnt/A/qiyh/2022/Pyrender/pyrender-master/pyrender/offscreen.py", line 149, in _create
    self._platform.init_context()
  File "/mnt/A/qiyh/2022/Pyrender/pyrender-master/pyrender/platforms/egl.py", line 177, in init_context
    assert eglInitialize(self._egl_display, major, minor)
  File "/mnt/A/qiyh/anaconda3/envs/pytorch120/lib/python3.6/site-packages/OpenGL/platform/baseplatform.py", line 409, in __call__
    return self( *args, **named )
  File "/mnt/A/qiyh/anaconda3/envs/pytorch120/lib/python3.6/site-packages/OpenGL/error.py", line 232, in glCheckError
    baseOperation = baseOperation,
OpenGL.error.GLError: GLError(
        err = 12289,
        baseOperation = eglInitialize,
        cArguments = (
                <OpenGL._opaque.EGLDisplay_pointer object at 0x7ff272c67d08>,
                c_long(0),
                c_long(0),
        ),
        result = 0
)

I have googled this problem but have no solution yet. Could you give some advice about how to validate the GL-related environment? My machine is a Ubuntu 16.04 server. Many thanks!

[JiayuYANG]

Hi,

I'm not familiar to egl so I can only provide limited help.

The egl we are using in Ubuntu is often provided by Nvidia's driver if I understand correctly. In here people mentioned of updating driver to solve this issue. Others also said you might need to update to Ubuntu 18.xx to solve this... Might be good to try these? Or just find another server with different system or driver and see if it work?

Btw, you can try to switch to osmesa instead of egl. Osmesa use CPU only and might be easier to setup.

Jiayu.

[XYZ-qiyh]

Thanks for your suggestions!

[XYZ-qiyh]

Hello, @JiayuYANG Now I have solved the problem about the EGL, and the demo code of OffscreenRenderer could work fine. But I'm confused about the transformation of camera pose between the OpenGL and the DTU dataset. https://github.com/JiayuYANG/Self-supervised-CVP-MVSNet/blob/794cc021d970d53dc75b11ad8f532831e8509cfa/self_training/pseudo_render.py#L108-L116

[XYZ-qiyh]

Hello, @JiayuYANG Now I have solved the problem about the EGL, and the demo code of OffscreenRenderer could work fine. But I'm confused about the transformation of camera pose between the OpenGL and the DTU dataset.

https://github.com/JiayuYANG/Self-supervised-CVP-MVSNet/blob/794cc021d970d53dc75b11ad8f532831e8509cfa/self_training/pseudo_render.py#L108-L116

Solved! https://github.com/colmap/colmap/issues/704#issuecomment-954161261

[zhuixunforever]

Hi, @JiayuYANG , sorry to bother you again. I have some problems with the underlying engine of the pyrender. When I run the offscreen rendering demo using EGL, it raises an error like this :

  File "offscreen_render.py", line 26, in <module>
    r = pyrender.OffscreenRenderer(400, 400)
  File "/mnt/A/qiyh/2022/Pyrender/pyrender-master/pyrender/offscreen.py", line 31, in __init__
    self._create()
  File "/mnt/A/qiyh/2022/Pyrender/pyrender-master/pyrender/offscreen.py", line 149, in _create
    self._platform.init_context()
  File "/mnt/A/qiyh/2022/Pyrender/pyrender-master/pyrender/platforms/egl.py", line 177, in init_context
    assert eglInitialize(self._egl_display, major, minor)
  File "/mnt/A/qiyh/anaconda3/envs/pytorch120/lib/python3.6/site-packages/OpenGL/platform/baseplatform.py", line 409, in __call__
    return self( *args, **named )
  File "/mnt/A/qiyh/anaconda3/envs/pytorch120/lib/python3.6/site-packages/OpenGL/error.py", line 232, in glCheckError
    baseOperation = baseOperation,
OpenGL.error.GLError: GLError(
        err = 12289,
        baseOperation = eglInitialize,
        cArguments = (
                <OpenGL._opaque.EGLDisplay_pointer object at 0x7ff272c67d08>,
                c_long(0),
                c_long(0),
        ),
        result = 0
)

I have googled this problem but have no solution yet. Could you give some advice about how to validate the GL-related environment? My machine is a Ubuntu 16.04 server. Many thanks!

Sorry to bother you, I have the same problem but in Ubuntu18.04, how did you solve it?

[XYZ-qiyh]

Hi, @zhuixunforever In my opinion, the problem is independent of the version of Ubuntu OS. In my lab, we have two machines. One is Ubuntu 16.04 with GTX2080 GPU cards, another is Ubuntu 18.04 with RTX3090 cards. The pyrender demo works on the latter machine (using EGL). I guess the problem is related to Nvidia's driver as Jiayu said, but I still don't know the specific relation between EGL and Nvidia's driver.

At last, don't forget the line os.environ["PYOPENGL_PLATFORM"] = "egl" in your python script.

[zhuixunforever]

@XYZ-qiyh Thank you for your reply! I will try to change the Nvidia's driver. By the way, I add the line os.environ["PYOPENGL_PLATFORM"] = "egl"