VR/AR-mode possible with this UE5 plugin

[FrankSpalteholz]

Dear Stereolabs-team, comming from Unity and switched over to UE i was wondering if this plugin also supports the VR/AR mode like in Unity? Thank you very much!

[SLJLMacarit]

Hello @FrankSpalteholz , thanks for reaching out!

Unfortunately, this plugin does not support AR/VR in HMDs like our Unity plugin, and there is no short-term plan to support it. Best regards,

Jean-Loup

[FrankSpalteholz]

Hey Jean-Loup! That's REALLY sad since I'm a big fan of the stereo-passthrough feature in Unity. Anyways. If I were able to write my own shader (seperating left and right) would it be possible to at least get the stereo-image-pair out of the plugin? In the initializer I can obviously just decide between left or right. Thx again

[Bvallon-sl]

Hi Frank,

Actually, our old UE plugin supported AR/MR features but it required to use a custom version of the Engine, especially for the shadows/lights parts. When we updated the plugin to UE5, we decided not to use a Custom engine anymore. Except for features that required custom modifications of the engine, all the old VR-related code is still present in the plugin but it has been commented out and considered "depreciated". All this part of the plugin will be removed in the next version as we decided not to continue supporting this feature in our UE plugin. For applications that require passthrough, we recommend using Unity.

May I ask you why you chose to use Unreal Engine instead of Unity?

Benjamin V

[FrankSpalteholz]

Dear Benjamin, that's good news actually. Can you point out which "special" version of UE4 you are refering to? We are currently working on a quite complex Multiplayer-VR-experience based on OptiTrack, using this for fusing all players, together with custom robotics, programm-able motion-seats, DMX-hardware and a lot more. I was hoping to add AR on top (with your excellent stereo-cams) by tracking a huge "cobot" with a StereoLabs-Cam attached to it and using OptiTrack for general tracking the head-set and the robot (which i already implemented) -> controlling then the motionseat with the tracking-data coming from the robot (so that the user "thinks" he/she is sitting on the robot) and adding then an extra AR-layer with a StereoLabs-cam. Because we are still on UE4.27.2 I wouldn't mind using an older plugin-version. (Although it would be a killer performance boost by using the GMSL2 feature with on of the new cams -> which i guess won't work with the older version) Anyways. All I "need" for now would be a proper stereo-pass-through-vision cos your system gives me all that out of the box without developing a stereo-cam-pair for pass-through myself (which i did several times so i really appreciate your excellent hardware with a quality better than what i was ever able to achieve) Add: And to answer your question regarding Unity vs Unreal. We aleady have a working Unreal-application so it would make no sense to switch to Unity + since I was switching to Unreal I'd hardly be willing to ever go back to Unity. Having the possibility to code in C++, the visual quality, the easy OptiTrack-implementation and most of all the possibilities for realtime-production and character-developement is something that Unity can't give me) Thank you so much for your fast reply and support!

[Bvallon-sl]

I would not recommend using the old plugin version at all as it is only compatible with UE4.21 and old SDK versions.

Most of the stereo passthrough-related code is still present in the current version of the plugin but has been disabled.

You can try to re-enable it by using this version of the plugin: https://github.com/stereolabs/zed-UE5/releases/tag/v1.2 and modify this line of code to re-enable it: https://github.com/stereolabs/zed-UE5/blob/v1.2/Plugins/Stereolabs/Source/ZED/Private/Core/ZEDPlayerController.cpp#L481.

However, you should be aware this feature has not been tested and is probably not fully functional. It will also be completely removed in the next version. Therefore, we will not provide any support for this feature in the future but I guess it can be a good start if you really need it for your project.

I'm sorry we can't provide you with a better solution but we decided to focus our efforts on Unity for the stereo passthrough (even if you project sounds super cool).

[FrankSpalteholz]

That's fine. I'll give it a try with the latest plugin then and see how far i can get. This is nothing we consider as "has to be production ready" and is more a internal prototype from us. Thank you so much for the support and if it's "ok" for you I'd really appreciate asking some more questions regarding this topic after getting a bit further practically.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days

[Xujiufeng]

Hello @Bvallon-sl UE5 MR Project missing shadow .we are developing a project using ue5, but the provided plug-in cannot produce shadows. But this function is very useful to us, and it also changed your code. But I don’t know how to add it. Hopefully this feature will continue to work.

[Bvallon-sl]

The UE5 project does not support this feature anymore. This required custom modifications of the Engine itself, which we no longer do.

Sorry for the inconvenience.

[Xujiufeng]

Our development team is based on ue5 and lacks unity developers. I also looked for many solutions online, but unfortunately I still couldn't solve the shadow problem. Is it possible to provide a way for us to learn? grateful!

[Bvallon-sl]

Our old plugin (for UE 4.21) is still available here : https://github.com/stereolabs/zed-unreal-plugin

You can take a look at it to see how we did it but you should know this plugin is obsolete and we do not provide support for it anymore.

[Xujiufeng]

4.21 custom distribution, the download link has expired, prompting 404.

[Bvallon-sl]

You need to follow there steps to have access to Unreal engine source code : https://www.unrealengine.com/en-US/ue-on-github

[Xujiufeng]

What is the corresponding cuda sdk version?