search

carla-simulator / carla

Open-source simulator for autonomous driving research.
http://carla.org
MIT License
11.2k stars 3.63k forks source link

UE4 initialization issue with VirtualGL #3423

Closed squizz617 closed 3 years ago

squizz617 commented 4 years ago

Hi. I've been experiencing severe slowdown of the initialization of CarlaUE4.uproject when being operated with VirtualGL.

Normally, e.g., on a machine with a physical monitor, after executing make launch, a number of processes such as ShaderWorkerCompiler are spawned and use 100% of CPU to do the initial compilation of shaders. I was able to see the UE4Editor GUI within a few minutes on moderately powerful machines; one with i7-4790K + GTX 1070ti and another with i7-8850H + Quatro P2000.

I have different setups where CARLA runs on headless servers (i.e., w/o a monitor). Things are different here. After setting up VirtualGL and VNC server to remotely access the screen, I've changed RHI option to -opengl and executed vglrun make launch. For some reason, none of the previously shader compiler processes are spawned, and the CPU usage is very low as if it's not doing any work. As a result, the initialization is prohibitively slower (usually stuck at 45% for days), even on dedicated server machines; one with i7-9700K + GTX 1080 ti, and another with Xeon Silver 4216 + Quatro RTX 8000. The only difference is the use of VirtualGL along with vncserver.

Could someone provide a solution for this issue?

qhaas commented 3 years ago

It will recompile everything if you previously ran Vulkan and are now using OpenGL...

A few ideas, assuming you have already ran vglrun glxgears + nvidia-smi and/or vglrun glxinfo to confirm VirtualGL is configured properly and using the GeForce instead of the 'fallback', which is far slower than using the remote GPU...

Change RHI to -opengl4 instead of just -opengl and/or place vglrun into the actual call to UnrealEditor instead of preceding the make call.

squizz617 commented 3 years ago

@qhaas Thanks a lot for the comment! I found that when vglserver was being configured, it failed to remove nvidia kernel modules, resulting in a permission issue. Manually rmmoding nvidia_drm, nvidia_modeset, nvidia_uvm, and nvidia, and then configuring vglserver (/opt/VirtualGL/bin/vglserver_config) fixed the aforementioned problem, and I could successfully launch UE4.

qhaas commented 3 years ago

Subtle! If reproducible, might be worth creating a minimally reproducible example of this behavior and reporting it to VirtualGL.

squizz617 commented 3 years ago

Yes, will do that!

And some additional information for those who might come across similar issues.. If the machine has one of the recent nvidia drivers installed, you might see that the frame rate always drops to 1 fps a few minutes after starting the machine, like shown in the image below: 1fps

This is most likely because of the "HardDPMS" feature, which mistakenly starts a power saving mode on the virtual monitor. Disable it by opening the xorg configuration file (/etc/X11/xorg.conf) and (1) changing line Option "DPMS" to Option "DPMS" "false" in the "Monitor" section, and also (2) adding Option "HardDPMS" "false" in the "Device" and "Screen" sections. After restarting the X service, all the glitches were gone, and I could successfully run VirtualGL + CARLA.

Reference: