How to run DReyeVR on three connected moniter display.

[shh1v]

The idea is to not use a VR headset for now, and I would like to run the simulator on three connected mounted monitors. The front monitor would display the windshield field of view, and the side monitors would display the view of the side windows.

[shh1v]

I assume three viewports must be rendered? If yes, would it give better performance compared to VR?

[GustavoSilvera]

Hmm. The whole point of DReyeVR is to use Carla in VR, so it is not really designed at all with the intention of being run on three displays.

This could be possible by looking further into UE4 nDisplay technology, but we have no knowledge of whether or not it would be successful with Carla/DReyeVR.

Also the performance would probably be similarly bad since the main performance killer is the independent rendering needed by each viewpoint (in VR you need 2: one for each eye. In nDisplay you need n, one for each display). But theoretically it could be easier to stack GPU's together and have each GPU render one display when using nDisplay, so you could maybe maybe get better parallelism that way (eg. using three RTX 3090's for three displays rather than one 3090 for VR).

As an aside, I would definitely recommend checking our @brifsttar's great blog where he discusses nDIsplay, VR, Carla, and more related to driving simulation.

[shh1v]

I realize that DReyeVR is not designed for multiple monitors display and that I will have to invest a significant(?) amount of time to implement this. However, we are really pushing for the completion of the study and submission to a conference. So, we are planning on testing multiple approaches at the same time, i.e., VR headset (with top-notch PC specs) and multiple monitors displays as a backup approach, and see what works out well. If VR works well, we plan to use that. However, if using a VR headset is creating more problems (ex: motion sickness) than it solves (ex: immersive experience), then it is not worth it to invest so much to get VR running. Thank you for such a detailed answer, and for mentioning further resources.

[brifsttar]

Some labs are using Carla on multiple monitors, like LIST in Luxembourg (video) or IBV in Spain (article). You can try to get in touch with them.

Depending on the field of view covered by the 3 monitors, you can fake it by configuring the main window to be extra wide. However, this obviously is not a valid projection and has a high performance cost, but I think it's what IBV is (was) doing. A few years ago on Carla's Discord (link), someone from LIST mentioned that they were prototyping their Arena with this:

My first attempt was to use the Carla python API to get 3 120deg Cameras side-by-side in the pygame view - which worked in theory - but was way to slow.

But as @GustavoSilvera mentioned, nDisplay is the correct answer to this problem, as it's exactly designed for such use cases. But I have no clue on how much work is needed to get that running in DReyeVR/Carla, I just know how to make it work in vanilla UE.

[shh1v]

Thank you so much for the insights @brifsttar and @GustavoSilvera. Based on your arguments, I think I will go with implementing ndisplay for DReyeVR.

[FBanani]

Thank you so much for the insights @brifsttar and @GustavoSilvera. Based on your arguments, I think I will go with implementing ndisplay for DReyeVR.

Hi @shh1v ,

I also want to run this driving simulator on three connected monitor displays. Have you made any progress?

[shh1v]

Hello @FBanani, hope you are doing well. We figured that implementing Unreal Engine's ndisplay framework was not viable due to lack of computational and time resources. Therefore, we changed the settings (see here) to launch as windowed full screen and adjusted some parameters such as FOV. The final result was pretty good for our need. I have attached a video of our setup (disregard the moving green dot, it just our eye-tracking implementation).

https://github.com/HARPLab/DReyeVR/assets/76883453/ff3b5ab3-bc14-46b3-8f36-85d496c435ad

Below is my camera parameters (in config file):

[CameraParams]
FieldOfView=120.0       # horizontal field of view (only in stereo camera => NOT VR)
ScreenPercentage=100   # 100% is native resolution, increase for supersampling, decrease for subsampling
# all the intensities range from [0 (off) to 1 (maximum)]
MotionBlurIntensity=0  # how much motion blur in the camera
VignetteIntensity=0    # how intense the vignetting is (darkened corners)
BloomIntensity=0       # how intense the bloom is
SceneFringeIntensity=0 # how intense the SceneFringe is
LensFlareIntensity=0   # how intense the lens flares are
GrainIntensity=0       # how intense the grain is

[FBanani]

Thank you so much. I will try it.

On Wed, May 1, 2024 at 8:41 PM Shiv Patel @.***> wrote:

[Caution: Email from External Sender. Do not click or open links or attachments unless you know this sender.]

Hello @FBanani https://github.com/FBanani, hope you are doing well. We figured that implementing Unreal Engine's ndisplay framework was not viable due to lack of computational and time resources. Therefore, we changed the settings (see here https://forums.unrealengine.com/t/packaged-game-starts-fullscreen/399257) to launch as windowed full screen and adjusted some parameters such as FOV. The final result was pretty good for our need. I have attached a video of our setup (disregard the moving green dot, it just our eye-tracking implementation).

https://github.com/HARPLab/DReyeVR/assets/76883453/ff3b5ab3-bc14-46b3-8f36-85d496c435ad

Below is my camera parameters (in config file):

[CameraParams] FieldOfView=120.0 # horizontal field of view (only in stereo camera => NOT VR) ScreenPercentage=100 # 100% is native resolution, increase for supersampling, decrease for subsampling

all the intensities range from [0 (off) to 1 (maximum)]

MotionBlurIntensity=0 # how much motion blur in the camera VignetteIntensity=0 # how intense the vignetting is (darkened corners) BloomIntensity=0 # how intense the bloom is SceneFringeIntensity=0 # how intense the SceneFringe is LensFlareIntensity=0 # how intense the lens flares are GrainIntensity=0 # how intense the grain is

— Reply to this email directly, view it on GitHub https://github.com/HARPLab/DReyeVR/issues/43#issuecomment-2089344536, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXQ7V2MQX2UTF5TUJXDDW3LZAGDUJAVCNFSM53TZ6SGKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBYHEZTINBVGM3A . You are receiving this because you were mentioned.Message ID: @.***>