search

CesiumGS / cesium-unreal

Bringing the 3D geospatial ecosystem to Unreal Engine
https://cesium.com/platform/cesium-for-unreal/
Apache License 2.0
905 stars 288 forks source link

Plugin doesn't work on HoloLens #502

Open argallegos opened 3 years ago

argallegos commented 3 years ago

Possibly related to the android build issues (https://github.com/CesiumGS/cesium-unreal/issues/190 https://github.com/CesiumGS/cesium-unreal/issues/472 ), a user on the forum reported issues packaging for HoloLens. Another forum user reported similar issues with launching a packaged project, indicating that the problem arose after adding Cesium World Terrain.

I don't think Hololens has been extensively tested with the plugin, but as far as I know it should work in general. Please add any relevant logs or errors to this issue.

drakelzh commented 3 years ago

According to my issue in https://community.cesium.com/t/does-this-plugin-support-hololens-platform-packaging/14095/3, I post my packaging record here. Although there is no error, there are some warnings.

Package.txt

getinthedamnbox commented 3 years ago

One more bug report regarding the HoloLens: https://community.cesium.com/t/no-terrain-shown-on-hololens-2/14250

I'm going to try to run a simple Cesium for Unreal level on the HoloLens 2, and I'll post my findings in this issue.

getinthedamnbox commented 3 years ago

Documenting what I've tried and what the results were, even though I haven't been able to figure out the problem:

There are at least four ways of testing HoloLens 2 applications:

  1. Install package to HoloLens 2 [link]
  2. Install package to HoloLens 2 Emulator [link]
  3. Deploy to device from Unreal editor [link]
  4. Use Holographic Remoting Player [link]

I was able to run the application with 1 and 2. More on that in a moment.

For 3, I get this error in Unreal:

LogPlayLevel: Warning: WARNING: Project will be packaged to support deployment to remote HoloLens device http://169.254.220.62/.
LogPlayLevel: Error: ERROR: Device connection failed: Requesting operating system information,

I know Unreal can talk to the device, because I can turn it off or restart it through Unreal. It fails to deploy the application, though.

For 4, I can't launch Holographic Remoting. I get this error:

Initialization Failure
Failed to initialize. Please make sure that Windows Holographic is installed on your system. Windows Holographic will be installed automatically when you attach your Head-mounted Display.

I haven't been able to figure out why Windows thinks Windows Holographic isn't installed. I'll start by trying to update Windows to get version 21H1, but I don't know how long that update will take, so I'm not doing it right now.

Back to 1 and 2 (testing on the HoloLens and Emulator):

Just as forum users described, I can see non-Cesium Actors in the level just fine, but I can't see any tilesets. I know from printing messages to the screen that the constructors and BeginPlay() functions of ACesium3DTileset and ACesiumGeoreference aren't being called on the HoloLens or the Emulator.

The process of retrieving logs can be mildly finicky, but it eventually works. Here are the errors I got on the HoloLens:

[2021.07.01-19.29.00:238][  0]LogStreaming: Error: Couldn't find file for package /Script/CesiumRuntime requested by async loading code. NameToLoad: /Script/CesiumRuntime
[2021.07.01-19.29.00:238][  0]LogStreaming: Error: Found 1 dependent packages...
[2021.07.01-19.29.00:238][  0]LogStreaming: Error:   /Game/Main
[2021.07.01-19.29.00:238][  0]LogPackageName: Error: DoesPackageExist: DoesPackageExist FAILED: '/CesiumForUnreal/CesiumCreditSystemBP' is not a standard unreal filename or a long path name. Reason: Path does not start with a valid root. Path must begin with: '/Engine/', '/Game/', '/WindowsMixedReality/', '/Takes/', '/Paper2D/', '/AnimationSharing/', '/GeometryMode/', '/SpeedTreeImporter/', '/DatasmithContent/', '/ChaosClothEditor/', '/GeometryProcessing/', '/GeometryCollectionPlugin/', '/ChaosSolverPlugin/', '/ChaosNiagara/', '/Niagara/', '/MotoSynth/', '/MediaCompositing/', '/Synthesis/', '/AudioSynesthesia/', '/WindowsMixedReality/', '/Takes/', '/Paper2D/', '/AnimationSharing/', '/GeometryMode/', '/SpeedTreeImporter/', '/DatasmithContent/', '/ChaosClothEditor/', '/GeometryProcessing/', '/GeometryCollectionPlugin/', '/ChaosSolverPlugin/', '/ChaosNiagara/', '/Niagara/', '/MotoSynth/', '/MediaCompositing/', '/Synthesis/', '/AudioSynesthesia/', '/WindowsMixedReality/', '/Takes/', '/Paper2D/', '/AnimationSharing/', '/GeometryMode/', '/SpeedTreeImporter/', '/DatasmithContent/', '/ChaosClothEditor/', '/GeometryProcessing/', '/GeometryCollectionPlugin/', '/ChaosSolverPlugin/', '/ChaosNiagara/', '/Niagara/', '/MotoSynth/', '/MediaCompositing/', '/Synthesis/', '/AudioSynesthesia/', '/Config/', '/Script/', '/Extra/', '/Memory/', or '/Temp/'
[2021.07.01-19.29.00:238][  0]LogStreaming: Error: Couldn't find file for package /CesiumForUnreal/CesiumCreditSystemBP requested by async loading code. NameToLoad: /CesiumForUnreal/CesiumCreditSystemBP
[2021.07.01-19.29.00:238][  0]LogStreaming: Error: Found 1 dependent packages...
[2021.07.01-19.29.00:238][  0]LogStreaming: Error:   /Game/Main
[2021.07.01-19.29.00:248][  0]LogStreaming: Error: ****DumpDependencies [Dependencies]:
[2021.07.01-19.29.00:248][  0]LogStreaming: Error:     Export 6 /Game/Main.Main:PersistentLevel.Cube
[2021.07.01-19.29.00:248][  0]LogStreaming: Error:     Linker is ../../../CesiumHoloLens/Content/Main.umap
[2021.07.01-19.29.00:248][  0]LogStreaming: Error:         Dep C_BEFORE_S Export     8    /Game/Main.Main:PersistentLevel.CesiumGeoreferenceDefault     (class CesiumGeoreference)
[2021.07.01-19.29.00:248][  0]LogStreaming: Error:         Dep C_BEFORE_S Export     7    /Game/Main.Main:PersistentLevel.CesiumCreditSystemDefault     (class CesiumCreditSystemBP_C)
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:         Dep C_BEFORE_S Export     6    /Game/Main.Main:PersistentLevel.Cube.TileSet     (class Cesium3DTilesetRoot)
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:         Dep S_BEFORE_C Import    12   /Script/CesiumRuntime.Cesium3DTileset
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:         Dep S_BEFORE_C Import     8   /Script/CesiumRuntime.Default__Cesium3DTileset
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:         Dep S_BEFORE_C Import     9   /Script/CesiumRuntime.Default__Cesium3DTileset:TileSet
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:         Dep C_BEFORE_C Export    14    /Game/Main.Main:PersistentLevel     (class Level)
[2021.07.01-19.29.00:253][  0]LogStreaming: Error: Missing Dependency, request for /Script/CesiumRuntime.Cesium3DTileset but it hasn't been created yet.
[2021.07.01-19.29.00:253][  0]LogStreaming: Error: Could not find class Cesium3DTileset to create Cube
[2021.07.01-19.29.00:253][  0]LogStreaming: Error: ****DumpDependencies [Dependencies]:
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:     Export 7 /Game/Main.Main:PersistentLevel.Cube.TileSet
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:     Linker is ../../../CesiumHoloLens/Content/Main.umap
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:         Dep C_BEFORE_S Import     8   /Script/CesiumRuntime.Default__Cesium3DTileset
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:         Dep S_BEFORE_C Import    13   /Script/CesiumRuntime.Cesium3DTilesetRoot
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:         Dep S_BEFORE_C Import     9   /Script/CesiumRuntime.Default__Cesium3DTileset:TileSet
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:         Dep C_BEFORE_C Export     5    /Game/Main.Main:PersistentLevel.Cube     (class Cesium3DTileset)
[2021.07.01-19.29.00:253][  0]LogStreaming: Error: Missing Dependency, request for /Script/CesiumRuntime.Cesium3DTilesetRoot but it hasn't been created yet.
[2021.07.01-19.29.00:253][  0]LogStreaming: Error: Could not find class Cesium3DTilesetRoot to create TileSet
[2021.07.01-19.29.00:253][  0]LogStreaming: Error: ****DumpDependencies [Dependencies]:
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:     Export 8 /Game/Main.Main:PersistentLevel.CesiumCreditSystemDefault
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:     Linker is ../../../CesiumHoloLens/Content/Main.umap
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:         Dep C_BEFORE_S Export    20    /Game/Main.Main:PersistentLevel.CesiumCreditSystemDefault.DefaultSceneRoot     (class SceneComponent)
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:         Dep S_BEFORE_C Import     2   /CesiumForUnreal/CesiumCreditSystemBP.CesiumCreditSystemBP_C
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:         Dep S_BEFORE_C Import    10   /CesiumForUnreal/CesiumCreditSystemBP.Default__CesiumCreditSystemBP_C
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:         Dep C_BEFORE_C Export    14    /Game/Main.Main:PersistentLevel     (class Level)
[2021.07.01-19.29.00:253][  0]LogStreaming: Error: Missing Dependency, request for /CesiumForUnreal/CesiumCreditSystemBP.CesiumCreditSystemBP_C but it hasn't been created yet.
[2021.07.01-19.29.00:253][  0]LogStreaming: Error: Could not find class CesiumCreditSystemBP_C to create CesiumCreditSystemDefault
[2021.07.01-19.29.00:253][  0]LogStreaming: Error: ****DumpDependencies [Dependencies]:
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:     Export 9 /Game/Main.Main:PersistentLevel.CesiumGeoreferenceDefault
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:     Linker is ../../../CesiumHoloLens/Content/Main.umap
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:         Dep S_BEFORE_C Import    14   /Script/CesiumRuntime.CesiumGeoreference
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:         Dep S_BEFORE_C Import    11   /Script/CesiumRuntime.Default__CesiumGeoreference
[2021.07.01-19.29.00:253][  0]LogStreaming: Error:         Dep C_BEFORE_C Export    14    /Game/Main.Main:PersistentLevel     (class Level)
[2021.07.01-19.29.00:253][  0]LogStreaming: Error: Missing Dependency, request for /Script/CesiumRuntime.CesiumGeoreference but it hasn't been created yet.
[2021.07.01-19.29.00:253][  0]LogStreaming: Error: Could not find class CesiumGeoreference to create CesiumGeoreferenceDefault
[2021.07.01-19.29.00:253][  0]LogStreaming: Error: Could not find outer CesiumCreditSystemDefault to create DefaultSceneRoot

Also worth noting that when packaging the application for HoloLens, I get the following warnings (similar to the log posted here):

UATHelper: Packaging (HoloLens):   LogInit: Display: LogSavePackage: Warning: /Game/Main imported Serialize:/CesiumForUnreal/CesiumCreditSystemBP.CesiumCreditSystemBP_C, but it was never saved as an export.
UATHelper: Packaging (HoloLens):   LogInit: Display: LogSavePackage: Warning: /Game/Main imported Serialize:/CesiumForUnreal/CesiumCreditSystemBP.Default__CesiumCreditSystemBP_C, but it was never saved as an export.
UATHelper: Packaging (HoloLens):   LogInit: Display: LogSavePackage: Warning: /Game/Main imported Serialize:/CesiumForUnreal/CesiumCreditSystemBP, but it was never saved as an export.
UATHelper: Packaging (HoloLens):   LogInit: Display: LogSavePackage: Warning: /Game/Main imported Serialize:/CesiumForUnreal/CesiumCreditSystemBP.CesiumCreditSystemBP_C:DefaultSceneRoot_GEN_VARIABLE, but it was never saved as an export.

I didn't attempt to change cesium-unreal/Source/CesiumRuntime/CesiumRuntime.Build.cs or cesium-unreal/extern/CMakeLists. That might be the obvious problem (HoloLens has an ARM architecture), but I would need help understanding how to fix it.

If anyone has a better grasp of the errors and warnings in the logs above, let me know.

kring commented 3 years ago

Thanks for the writeup @getinthedamnbox. I suspect if you scroll up further in the log, you'll see the Cesium for Unreal plugin isn't being loaded at all. Can you confirm that? I'm not really sure why that would be, though. You said the HoloLens is ARM. It's 64-bit ARM, right? Is it Android? Does it use the same toolchain as regular Android builds?

getinthedamnbox commented 3 years ago

I suspect if you scroll up further in the log, you'll see the Cesium for Unreal plugin isn't being loaded at all. Can you confirm that?

You're right, I don't see it listed as one of the mounted plugins. Here's the full log: CesiumHoloLens-backup-2021.07.01-19.29.14.log

And here's the log from packaging the application for HoloLens: CesiumHoloLens.log

You said the HoloLens is ARM. It's 64-bit ARM, right? Is it Android? Does it use the same toolchain as regular Android builds?

It's 64-bit ARM, but to be honest I don't know enough about toolchains to answer your other questions. What I know is that in order to package and deploy the application, I had to ensure Visual Studio Build Tools had the component MSVC v142 - VS 2019 C++ ARM64 build tools (v14.26) installed.

kring commented 3 years ago

Thanks @getinthedamnbox. Yeah that definitely doesn't sound like "normal" Android, at least. So we probably need a special build of cesium-native.

argallegos commented 3 years ago

@kring Alban had some insights:

If you plan to run on the device, you need to have an ARM64 build of Cesium.

When you run the application using holoremoting, it basically runs on your computer, and streams the 3d render in realtime like a video. On the other hand, the HL2 sends its interactions to the device, and you close the loop... In that mode (Tethered or not - I prefer to use Wifi), the app has to be packaged in Win64, so it's very easy to do so using the current Cesium plugin. (I did an app for GTC showcasing that).

But it you plan to deploy and run it "on" the device (meaning without any computer to generate the 3D content), your app needs to be packaged for Hololens2, that expects an ARM64 build. It's fine with most UE libraries, but any included external lib needs an ARM64 version.

I had to do that to port the GeoReferencing plugin I wrote for 4.27. It relies on PROJ, so I had to build PROJ for ARM64 to make it run. Fortunately, it was not that hard when using vcpkg toolchain.

Then, as usual, you need to edit your 3rd party libs build.cs to point to the right libs, with a distinction on architecture. See this sample file for the proj.build.cs


else if (Target.Platform == UnrealTargetPlatform.HoloLens)
{
if (Target.WindowsPlatform.Architecture == WindowsArchitecture.x64) // emulation target, bBuildForEmulation
{
PublicSystemIncludePaths.Add(Path.Combine(ModuleDirectory, VcPkgInstalled, "overlay-x64-uwp", "include"));
            string LibPath = Path.Combine(ModuleDirectory, VcPkgInstalled, "overlay-x64-uwp", "lib");
            PublicAdditionalLibraries.Add(Path.Combine(LibPath, "proj.lib"));
            PublicAdditionalLibraries.Add(Path.Combine(LibPath, "sqlite3.lib"));
        }
        else if (Target.WindowsPlatform.Architecture == WindowsArchitecture.ARM64) // device target, bBuildForDevice
        {
            PublicSystemIncludePaths.Add(Path.Combine(ModuleDirectory, VcPkgInstalled, "overlay-arm64-uwp", "include"));

            string LibPath = Path.Combine(ModuleDirectory, VcPkgInstalled, "overlay-arm64-uwp", "lib");
            PublicAdditionalLibraries.Add(Path.Combine(LibPath, "proj.lib"));
            PublicAdditionalLibraries.Add(Path.Combine(LibPath, "sqlite3.lib"));
        }
        else
        {
            throw new System.Exception("Unknown architecture for HoloLens platform!");
        }
    }

> Don't be afraid by the "uwp". This is specific to PROJ to allow it to access its data file on a sandboxed device.
> Proj.Build.cs 

// Copyright Epic Games, Inc. All Rights Reserved.

using System.IO; using UnrealBuildTool;



> If you want to pick up some code, it's on UE Github there : https://github.com/EpicGames/UnrealEngine/tree/4.27/Engine/Plugins/Runtime/GeoReferencing/Source/GeoReferencing
> Having a build of cesium for ARM64 might be complicated, because you are using lots of external libs.
> But once done, it will be great. I hope the HL2 GPU will be able to deal with this geometry, without raising too much the screen space error value.
ahsan4313 commented 1 year ago

any update regarding the issue

kring commented 1 year ago

No update. If someone wants to attempt this, follow the Developer Setup instructions, adjusting them to build cesium-native for the HoloLens platform, probably by defining a cmake toolchain. It is not currently on our roadmap, but we would welcome a pull request from the community.

ahsan4313 commented 1 year ago

Sure will try it. Will this work remotely without packaging?

ahsan4313 commented 1 year ago

image After running camke files for Cesium native. what steps i have to take for Hololens 2 plugin. please update me

ahsan4313 commented 1 year ago

image Got this html file running after following the process for cesium native. Now what to do?

kring commented 1 year ago

If we knew the exact steps, it would already be done. I'm afraid I don't know, and you'll have to figure it out, or wait for someone else to figure it out. From what I can see in your VS Code window, you're just building cesium-native for Windows, so that's not going to help. You need to build it for HoloLens by configuring a toolchain appropriately, but I don't know any more details than that.

I'm not sure what that Eigen3 page is. It's not part of cesium-native, but maybe one of our third-party libraries uses it indirectly.

ahsan4313 commented 1 year ago

Great. Just want to confirm is this for desktop hologram or android hologram?

Although i think its for hologram (doesn't matter platform) but want to clear ambiguity of my senior.

ahsan4313 commented 1 year ago

Can I access cesium for unreal plugin while connecting HoloLens remotely?

kring commented 1 year ago

Yes, Cesium for Unreal should already working with HoloLens remoting.

ahsan4313 commented 1 year ago

Great