RobotLocomotion / drake

Model-based design and verification for robotics.
https://drake.mit.edu
Other
3.19k stars 1.24k forks source link

[render_gl] Remove X server requirement #21050

Open pathammer opened 4 months ago

pathammer commented 4 months ago

Is your feature request related to a problem? Please describe.

According to @jwnimmer-tri, there's no reason for an X server to be a strict requirement for renderering. It makes it harder to deploy to headless environments for containerized environments.

Describe the solution you'd like

RenderEngineGL could use EGL to create it's OpenGL context.

~VTK can automatically use EGL if built with that option.~ Edit: See #21700.

Describe alternatives you've considered

I considered using xvfb but that does non-HW-accelerated rendering and would be a bottleneck.

jwnimmer-tri commented 4 months ago

FYI on a related topic...

As of Bazel 7.0, to access a real Xorg server during Bazel tests downstream projects must add this line in their .bazelrc:

build --sandbox_add_mount_pair=/tmp/.X11-unix

Otherwise, you'll see an failure such as DRAKE_THROW_UNLESS(display != nullptr); around internal_opengl_context.cc:85.

This is due to the following change in Bazel 7: _--incompatible_sandbox_hermetic_tmp is enabled by default_.

jwnimmer-tri commented 3 months ago

I tried some prototyping today...

I used this WIP branch that switches VTK rendering to use EGL instead of GLX. I built a wheel file, installed it on Deepnote, and did apt install libegl-mesa0 instead of install_xvfb. I ran rendering_multibody_plant.ipynb and it worked fine. Thus, EGL seems like a viable option for CPU-based rendering without an X server.

As I understand it, there are ways to teach EGL to use the GPU, but I haven't looked into that specifically yet.

When using EGL, the camera option to show_window = True seemingly has no effect, at least not in my WIP branch. Possibly some device configuration is necessary, or maybe I need to be using Wayland instead of Xorg. Also, most of the VTK render engine tests pass, but two cases fail (for me).

Assuming that EGL can't be made strictly better than GLX, the next steps would be to see if we can compile both GLX and EGL support into Drake's VTK, and offer a RenderEngineVtkParams option for which one to use.

jwnimmer-tri commented 1 week ago

For our Kitware helpers -- let's say the first milestone on this issue is just RenderEngineVtk. We'll leave the RenderEngineGl for a second milestone.

jwnimmer-tri commented 1 week ago

Actually, I take that back. Let's use #21700 for the VTK option, and keep this ticket for the RenderEngineGl changes.