tacto headless setup errors

shivanimall commented 3 years ago

Note: I uncommented the following code: https://github.com/facebookresearch/tacto/blob/master/tacto/renderer.py#L21 which is equivalent to your suggestion on using EGL Mode

Below is the error I get when attempting to run tacto headless:

Traceback (most recent call last):
  File "demo_pybullet_digit.py", line 23, in main
    digits = tacto.Sensor(**cfg.tacto, background=bg)
  File "/viscam/u/small02/tacto/tacto/sensor.py", line 80, in __init__
    self.renderer = Renderer(width, height, background, config_path)
  File "/viscam/u/small02/tacto/tacto/renderer.py", line 77, in __init__
    self._init_pyrender()
  File "/viscam/u/small02/tacto/tacto/renderer.py", line 115, in _init_pyrender
    self.r = pyrender.OffscreenRenderer(self.width, self.height)
  File "/sailhome/small02/miniconda2/envs/tacto/lib/python3.8/site-packages/pyrender/offscreen.py", line 31, in __init__
    self._create()
  File "/sailhome/small02/miniconda2/envs/tacto/lib/python3.8/site-packages/pyrender/offscreen.py", line 149, in _create
    self._platform.init_context()
  File "/sailhome/small02/miniconda2/envs/tacto/lib/python3.8/site-packages/pyrender/platforms/egl.py", line 177, in init_context
    assert eglInitialize(self._egl_display, major, minor)
  File "/sailhome/small02/miniconda2/envs/tacto/lib/python3.8/site-packages/OpenGL/platform/baseplatform.py", line 402, in __call__
    return self( *args, **named )
  File "/sailhome/small02/miniconda2/envs/tacto/lib/python3.8/site-packages/OpenGL/error.py", line 228, in glCheckError
    raise GLError(
OpenGL.error.GLError: GLError(
    err = 12289,
    baseOperation = eglInitialize,
    cArguments = (
        <OpenGL._opaque.EGLDisplay_pointer object at 0x7f00379bbdc0>,
        c_long(0),
        c_long(0),
    ),
    result = 0
)

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

wx405557858 commented 3 years ago

Hi @shivanimall ,

Could you provide information about your OS, NVIDIA driver, cuda version? Just a quick check, in order to use EGL mode, it needs GPU acceleration. If no GPU available, it needs to be set for "OSMesa" backend for headless rendering.

shivanimall commented 3 years ago

+-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+

shivanimall commented 3 years ago

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.1 LTS Release: 20.04

wx405557858 commented 3 years ago

Hi @shivanimall ,

Got it, will this be helpful? https://github.com/mmatl/pyrender/issues/7 and one solution pointed to this link: https://github.com/mcfletch/pyopengl/issues/27

shivanimall commented 3 years ago

Hi @wx405557858 thanks for pointing those I followed the solution in link

but now getting the following error:

Python 3.8.8 (default, Apr 13 2021, 19:58:26) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tacto
pybullet build time: Jun  2 2020 06:49:02
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/viscam/u/small02/tacto/tacto/__init__.py", line 6, in <module>
    from .sensor import (  # noqa: F401
  File "/viscam/u/small02/tacto/tacto/sensor.py", line 18, in <module>
    from .renderer import Renderer
  File "/viscam/u/small02/tacto/tacto/renderer.py", line 28, in <module>
    import pyrender
  File "/sailhome/small02/miniconda2/envs/tacto/lib/python3.8/site-packages/pyrender/__init__.py", line 3, in <module>
    from .light import Light, PointLight, DirectionalLight, SpotLight
  File "/sailhome/small02/miniconda2/envs/tacto/lib/python3.8/site-packages/pyrender/light.py", line 10, in <module>
    from OpenGL.GL import *
  File "/sailhome/small02/miniconda2/envs/tacto/lib/python3.8/site-packages/OpenGL/GL/__init__.py", line 4, in <module>
    from OpenGL.GL.VERSION.GL_1_1 import *
  File "/sailhome/small02/miniconda2/envs/tacto/lib/python3.8/site-packages/OpenGL/GL/VERSION/GL_1_1.py", line 14, in <module>
    from OpenGL.raw.GL.VERSION.GL_1_1 import *
  File "/sailhome/small02/miniconda2/envs/tacto/lib/python3.8/site-packages/OpenGL/raw/GL/VERSION/GL_1_1.py", line 7, in <module>
    from OpenGL.raw.GL import _errors
  File "/sailhome/small02/miniconda2/envs/tacto/lib/python3.8/site-packages/OpenGL/raw/GL/_errors.py", line 4, in <module>
    _error_checker = _ErrorChecker( _p, _p.GL.glGetError )
AttributeError: 'NoneType' object has no attribute 'glGetError'

https://github.com/NixOS/nixpkgs/issues/76822 I found this issue but no fix pointed out in it? since they resolved it, let me know if I missed something.

wx405557858 commented 3 years ago

Hi @shivanimall ,

Any chance these might be helpful? Someone says changing the pyopengl version, or remove the current pyopengl and apt-get install python-opengl

https://github.com/MPI-IS/mesh/issues/23 https://githubmemory.com/repo/TimoBolkart/TF_FLAME/issues/10

shivanimall commented 3 years ago

Hello @wx405557858,

I could not exactly try the above solution since I do not have access to run apt-get on the server. But I tried installing directly from their source tar package and their github repo. I also tried changing the error_check value as suggested on a thread. none of those worked. continue to get the same error.

asked an admin to do apt-get install of this package. But in case, this doesn't work, what are the alternatives?

Thank you so much for your continued support with these.

wx405557858 commented 3 years ago

Hi @shivanimall ,

I see, the alternatives would be using osmesa, it might not as fast as egl, but should also work headless. Our circleci test is using osmesa headless: https://github.com/facebookresearch/tacto/blob/master/.circleci/config.yml

# packages for headless rendering 
apt-get install -y libgl1-mesa-glx libosmesa6 freeglut3 freeglut3-dev

Here's the instructions from pyrender: https://pyrender.readthedocs.io/en/latest/install/index.html

Here's an example: https://github.com/facebookresearch/tacto/blob/master/tests/test_render_from_depth_osmesa.py

shivanimall commented 3 years ago

thank you @wx405557858 I also wanted to note below: that my collaborator tried out an option using xvbf on a different GPU partition which we can request and it worked. I can ask him to post his solution below in case it may benefit the community.

so we finally have this working on the desired server, thank you so much again for your support with this.

wx405557858 commented 3 years ago

Hi @shivanimall ,

That's awesome!!! Really appreciate it if your teammate could post his solution! Thank you!

shivanimall commented 3 years ago

posting solution below:

0. used napoli gpu (telsa K40 worked)
created conda environment and installed all the required packages, then:
1. pip install pyrender
2. pip install pyopengl
3. use egl mode
5. Xvfb :99 -screen 0 640x480x24 &
6. export DISPLAY=:99

wx405557858 commented 3 years ago

Thank you! Appreciate it!

facebookresearch / tacto

tacto headless setup errors #21