search

VirtualGL / virtualgl

Main VirtualGL repository
https://VirtualGL.org
Other
701 stars 106 forks source link

VirtualGL segfaults when using OpenGL software render #239

Closed Xaraxia closed 1 year ago

Xaraxia commented 1 year ago

One of our academics found an interesting bug. When wrapping Matlab in vglrun, it works fine (so far, anyway) when you remember to tell it not to do software rendering. But if you run matlab with no arguments inside a vglrun-wrapped desktop environment, it defaults to OpenGL software render and you get the following as soon as you try to plot something:

Abnormal termination:
Segmentation violation

Current Thread: 'MATLAB' id 139801197995776

Register State (from fault):
  RAX = 0000000000000000  RBX = 00007f2a08b84890
  RCX = 0000000000000001  RDX = 00007f2600beea10
  RSP = 00007f2600becab8  RBP = 00007f2600becba0
  RSI = 00007f2600becb2c  RDI = 0000000000008d57

   R8 = 0000000000000000   R9 = 0000000000000001
  R10 = 00000000000002e2  R11 = 00007f2a07d71790
  R12 = 00007f2a08ba6be8  R13 = 0000000000706d10
  R14 = 0000000000000000  R15 = 00007f2a08b84b10

  RIP = 00007f2a072842fb  EFL = 0000000000010246

   CS = 0033   FS = 0000   GS = 0000

Stack Trace (from fault):
[  0] 0x00007f2a072842fb /sw/local/rocky8.6/noarch/rcc/software/MATLAB/R2022a6/sys/opengl/lib/glnxa64/libGL.so.1+03420923 glGetIntegerv+00000011
[  1] 0x00007f2a08b02781 /sw/auto/rocky8c/epyc3_l40/software/VirtualGL/3.1-GCC-11.3.0/lib64/libvglfaker.so+00616321
[  2] 0x0000000000706d10                                   <unknown-module>+00000000

We have a workaround (always disable software rendering), but I figured this is probably indicative of a bug somewhere, so I'd shoot it in your direction. It can be reliably reproduced.

dcommander commented 1 year ago

In general, you cannot use a software-only Mesa implementation (including the swrast driver) as a back end for VirtualGL. It doesn't work and never has. There isn't any point to doing that anyhow, since VirtualGL's sole purpose is to redirect rendering to a GPU. If you're going to run MATLAB with software OpenGL, then don't use VirtualGL. If you're going to run MATLAB with VirtualGL, then don't use software OpenGL. Our User's Guide says as much: https://rawcdn.githack.com/VirtualGL/virtualgl/3.1/doc/index.html#hd0015

Xaraxia commented 1 year ago

Should have checked the User Guide. Thanks for that, appreciate the fast response. Will add these instructions to our own user's guide.

dcommander commented 1 year ago

I'm not sure if this is still the case, but back in the day, MATLAB would incorrectly enable software OpenGL by default if it detected that the X server had an extension named VNC-EXTENSION. This caused it to always enable software OpenGL by default when running in a TurboVNC, TigerVNC, or RealVNC session, regardless of whether VirtualGL was used. IMHO, there is little or no justification these days for applications to provide their own OpenGL implementations. All actively supported Un*x distributions support OpenGL 4.6, to the best of my knowledge, and if for some reason your GPU driver doesn't (which isn't true for any GPU driver I'm aware of), then you can manipulate an environment variable to force the use of software OpenGL. If MathWorks insists on continuing to bundle its own OpenGL implementation, then they should at least change their detection logic so that software OpenGL isn't automatically enabled if VirtualGL is active. (VirtualGL can be detected by looking at the GLX vendor string.)