How to choose the correct enviroment for installation including all functions like eye tracking?

[wx983]

I have seen on the official document that to obtain the full fuctions, we should be based on Windows. MeanWhile, the official document also says WSL is needed. So whether to use WSL or Windows command to construct the DeyeVR?

[GustavoSilvera]

Hi, sorry for the confusion. We are working on improving the documentation with an upcoming release.

WSL used to be the required/recommended installation method on windows since we used bash scripts (.sh) to copy files over, but recently we revamped this installation method with python that is os-compatible with batch (windows shell) and unix (linux/mac). So now you do not need WSL and can do all your installation from the cmd (using make install and/or make check in the DReyeVR repo, just like with Carla).

It is true that the Eye-tracking and logitech-steering-wheel plugin currently only support Windows at the moment, so you'll need to install DReyeVR on a Carla installation running on Windows to enable these features. If you'd like to individually enable/disable these (and upcoming) features you can toggle these flags here (the defaults are enabling the eye tracking and logitech wheel)

[wx983]

Thank you for answering! In recent days, I'm trying to build carla on windows. Later on, I will install DeyeVR based on windows cmd completely rather than WSL.

[wx983]

Hi, I encounter some new issues but I'm still rather not sure about them after looking through large quantity of documents. As I mentioned in my last question before, I want to fulfill the complete functions of DeyeVR on windows. My problem is about Eye tracking part.

Problem 2-4 are mainly about the compatiability between UnrealEngine 4.26 and VRS Plugin.

1. Does Variable Rate Shading(VRS) is particularly necessary or not for eye tracking? I think it's yes but I'm not very sure.

2. If question 1 is essential, we use UnrealEngine 4.26 in DeyeVR, but here . Did you revise these files according to it or just ignore them?

3. There are several versions of VRS here, but 4.26 is just not in the lists of them. How to choose the right VRS Plugin version? I wonder how you choose them. Thanks very much!

4. According to [here](https://forum.htc.com/topic/7434-getting-started-with-vrs-foveated-rendering-using-htc-vive-pro-eye-unreal-engine/#:~:text=%5B2021%20%E5%B9%B4%202%20%E6%9C%88%209%20%E6%97%A5%E6%9B%B4%E6%96%B0%5D%EF%BC%9A%E5%A6%82%E6%9E%9C%E6%82%A8%E4%BD%BF%E7%94%A8%20UE%204.24.2%20%E6%88%96%204.23.1%20%E7%9A%84%E8%99%9A%E5%B9%BB%E5%BC%95%E6%93%8E%E5%88%86%E6%94%AF%EF%BC%8C%E5%88%99%E6%97%A0%E9%9C%80%E7%BC%96%E5%86%99%E4%BB%BB%E4%BD%95%E9%A2%9D%E5%A4%96%E4%BB%A3%E7%A0%81%EF%BC%8C%E6%82%A8%E5%8F%AF%E4%BB%A5%E5%BF%BD%E7%95%A5%E4%BB%A5%E4%B8%8B%E6%AD%A5%E9%AA%A4%E3%80%82%E6%82%A8%E4%B8%8D%E9%9C%80%E8%A6%81%E6%B7%BB%E5%8A%A0%20GetEyeGazeDirectionsNormalized()%20%E5%92%8C%E8%B0%83%E7%94%A8%20UpdateStereoGazeDataToFoveatedRendering()%EF%BC%8C%E6%82%A8%E5%8F%AA%E9%9C%80%E5%AE%89%E8%A3%85%20SRanipal%20%E8%BF%90%E8%A1%8C%E6%97%B6%EF%BC%8C%E5%B0%86%20SRanipal%20%E6%8F%92%E4%BB%B6%E5%AF%BC%E5%85%A5%E6%82%A8%E7%9A%84%E6%B8%B8%E6%88%8F%E9%A1%B9%E7%9B%AE%E5%B9%B6%E5%9C%A8%20VRSSettings%20%E4%B8%AD%E5%90%AF%E7%94%A8%E7%9C%BC%E5%8A%A8%E8%BF%BD%E8%B8%AA%EF%BC%8C%E7%84%B6%E5%90%8E%20VRSPlugin%20%E5%B0%86%E8%87%AA%E5%8A%A8%E4%BB%8E%E5%BC%95%E6%93%8E%E8%8E%B7%E5%8F%96%E6%B3%A8%E8%A7%86%E6%95%B0%E6%8D%AE%E6%82%A8%E5%8F%AA%E9%9C%80%E9%85%8D%E7%BD%AE%E8%AE%BE%E7%BD%AE%E5%8D%B3%E5%8F%AF%E3%80%82%5D), it seems that we need to add some extra codes. I don't know if we need to do so.

Best wishes.

[GustavoSilvera]

Hi,

Good questions! I'll answer in order:

1. Variable Rate Shading is a technique to modify (downsample or upsample) the "quality" of the rendering shader in select areas of the scene depending on various factors: fixed, content-aware, eye gaze. We are primarily interested in this to achieve extra performance by reducing the quality (and rendering cost) of the periphery which we can calculate using our eye gaze data. However, for DReyeVR's default Eye tracking, VRS (and foveated rendering) is NOT required. It is entirely an (optional) optimization process. In fact, it is currently broken in main but we have WIP development to get it functional again.
2. Question 1 is not essential, so you can skip all the Foveated-rendering notes. But we did have to use a modified UE4 fork from CarlaUnreal, for best results on 0.9.13 you should use the commit d40ec35474e8793b4eea60dba6c4f051186e458e.
3. I worked a bit on porting the ViveSR (from UE4.24) work to UE4.26 myself, but it is not ready for production yet. Currently we have seen positive results in the Editor but not in Shipping build yet. Once this is working we'll ship this feature to all users.
4. To use DReyeVR normally (without Foveated Rendering) you don't need to use any of this extra code, all the important code is already provided by us. You only need to install the SRanipal (and optionally Logitech) plugin and use the CarlaUnreal fork described above. If you have any questions or are unable to download SRanipal plugin 1.3.1.1 or 1.3.3.0, contact us and we can provide you with a copy.

[wx983]

Thanks for answering in detailed! But I have one place I don't really understand. You mean, VRS has broken in 'main'?

[GustavoSilvera]

Currently, DReyeVR does not support VRS because the code changes to UE4.26 haven't been successful yet. I am personally working on this and it should be fixed in the near future. This is currently an optional feature that is disabled by default (see here) so end users don't have to worry about it.

[wx983]

Oh, good guide, thanks! On windows, the offical document says we need LogitechWheel and SRanipal Plugins as following: But it also points out that we need install UE4 SteamVR plugin.

So do we need to place the unzipped folder in carla/Unreal/CarlaUE4/Plugins/ as following? I think is yes.

[GustavoSilvera]

Hi, yes you should be able to install SteamVR from the steam application itself, and installing the steamvr unreal plugin is only required for Linux builds (which it looks like you are not using). With SRanipal (for eye tracking) and Logitech (for steering wheel) installed you should be able to select to "play in VR mode" in the editor once SteamVR is installed. For more help you can refer to the SetupVR.md guide.

[wx983]

Dear, I have installed DReyeVR successfully right now, and I'm able to play VR mode in UE4 Editor as fowllowing, using LogitechWheel to control vehicle and HTC Eye Pro device to view around:

But when I run CarlaUE4.exe -vr (from the x64 Visual C++ Toolset command prompt), it opens and collapses after a while. It runs(CarlaUE4.exe -vr) no problem before installing DReyeVR thoroughly. Is it a normal phenomenon for each DReyeVR installation?

[GustavoSilvera]

Hi, no, this is not normal behavior and there is definitely something wrong. You should not be seeing crashes at all in the shipping build (CarlaUE4.exe -vr). Can you check if you can find crash logs in the WindowsNoEditor folder after the crash?

[GustavoSilvera]

The log (CarlaUE4.log) files for shipping crashes should be in the WindowsNoEditor/CarlaUE4/Saved/Crashes/ directory.

Additionally, you can force the game to enable logging while in package/shipping mode by adding the following line to the Target file (in Carla/Unreal/CarlaUE4/Source/CarlaUE4.Target.cs) and rebuild. Then you should see logs in all your package builds located in C:\Users/%YOUR_USER_NAME%/AppData/Local/CarlaUE4/Saved/Logs/CarlaUE4.log

// to make it look like this:

public class CarlaUE4Target : TargetRules
{
    public CarlaUE4Target(TargetInfo Target) : base(Target)
    {
        Type = TargetType.Game;
        ExtraModuleNames.Add("CarlaUE4");
        bUseLoggingInShipping = true;//                 <--- added here
    }
}

See this guide for more help

[wx983]

Hi, Today, even in the UE4 Editor it doesn't work either. In other words, it also crashes after launching in UE4 Editor, not only in 'CarlaUE4.exe -vr' from x64 cmd. After rebuilding, I see the log file in 'C:\Users/%YOUR_USER_NAME%/AppData/Local/CarlaUE4/Saved/Logs/'. It was generated after executing 'CarlaUE4.exe -vr', reporting errors as following(line 477 to 492 && line 598 to 599):

It seems that some oponents are missing, what should I do next? Reinstall DReyeVR or something else?

Total log file is here. CarlaUE4.log

[GustavoSilvera]

Hm, I haven't seen this error message before. Did you run Update.bat on the vanilla Carla prior to installing DReyeVR?

[GustavoSilvera]

Also, which version of SRanipal are you using? Can you verify it is 1.3.3.0 rather than 1.3.6+

[GustavoSilvera]

You can explicitly download SRanipal 1.3.3.0 from this link: https://hub.vive.com/en-US/download/SDK-v1.3.3.0.zip

[wx983]

I verify I run Update.bat on pure Carla before installing DReyeVR. About next question, I use 1.3.6.6 rather than 1.3.3.0. Next I will change it to 1.3.3.0.

[wx983]

Hi, Is it(Bug 2) essential to fix in windows cmd(not WSL)？

If necessary, how can we do that? Should we run this before it? Because I don't see 'make patch-sranipal' in the installation command summary

[GustavoSilvera]

Yes, bug2 must be fixed. This can be run with make patch-sranipal CARLA=\PATH\TO\CARLA as defined here. Sorry for the confusing documentarion! We are working on improving this in the upcoming release.

(The patch_sranipal.sh is an outdated reference before we updated our build system to python+make)

[wx983]

Yes, bug2 must be fixed. This can be run with make patch-sranipal CARLA=\PATH\TO\CARLA as defined here. Sorry for the confusing documentarion! We are working on improving this in the upcoming release.

(The patch_sranipal.sh is an outdated reference before we updated our build system to python+make)

It doesn't matter. So we don't have to run 'make patch-sranipal' before running 'make patch-sranipal CARLA=../carla' in order to installing DReyeVR?

[GustavoSilvera]

The only diference is that passing in CARLA= as a parameter enables the patch to apply to a specific directory. But if you omit this (optional) parameter it will just follow your `%CARLA_ROOT% environment variable to find your Carla installation and Patch SRanipal once it is installed (in your Plugin directory).

So no, it doesn't matter as long as %CARLA_ROOT% (on Windows) is the path you installed Carla to

[wx983]

另外，您使用的是哪个版本的 SRanipal？你能确认它是 1.3.3.0 而不是 1.3.6+

Cheerfully, it works after changing SRanipal 1.3.6.6 to 1.3.3.0! Thanks!

[GustavoSilvera]

Great, yeah we've also seen unknown issues with SRanipal 1.3.6.+ so we recommend exactly 1.3.3.0

Is it working in both Editor and Package mode?

[wx983]

太好了，是的，我们也看到了 SRanipal 1.3.6.+ 的未知问题，所以我们推荐 1.3.3.0

它是否在编辑器和包模式下工作？

Yes, it works in both 'CarlaUE4.exe -vr' from x64 cmd on windows and Editor. Here is in 'CarlaUE4.exe -vr' from x64

Here is in Editor.

[wx983]

In addition, I feel performance in ' CarlaUE4.exe -vr' from x64' is much better and more fluent than in the Editor, although in Editor the performance is OK. Mine is NVIDIA 3080.

[GustavoSilvera]

Awesome, glad to hear it! We are working on updates and should have improved performance features coming up in the next release.

[wx983]

@GustavoSilvera Hello, recently I encountered one problem when installing DReyeVR. But I still can not find an suitable way to solve it. When running make package in x64 command line, it reports one strange error as the second picture shows. The physical memory of computer is enough and it seems the virtual memory is not enough, which puzzled me a lot.