Guide: How to make the Spectator-View work on Hololens 2 + Android

[ChristopherRemde]

Intro

After weeks of being stuck on error after error on this project, I wanted to share a complete guide on how to make this project work for Hololens 2 + Android, so that others don't have to spend this much time in developer hell.

This guide is split into two parts:

1: Building the MRSV-Unity Package

2: Building, deploying and using the package for Hololens 2 and Android.

If you want to skip building the Unity Package, I have provided a pre-build package here. You can download it & directly go to part 2!

Prerequesites

• Unity 2019.2.21 It is very important that you use this specific version of Unity, other Versions, even with only minor differences will make the builds fail!

  • Add the Android, UWP and Windows build packages to the Unity install
  • Select all options (Install JDK, NDK ect..) for the Android package

• Visual Studio Enterprise 2019

• Inside of Visual Studio Enterprise, you have to install:

  • .Net Desktop Development, C++ Desktop Development
  • Universal-Windows Platoform Development
  • Game Development with Unity
  • MSVC v142 - VS 2019 C++ ARM64 Buildtools (latest)
  • MSVC v142 - VS 2019 C++ ARM64 Libraries (latest)
  • MSVC v142 - VS 2019 C++ ARM Buildtools (latest)
  • MSVC v142 - VS 2019 C++ ARM Libraries (latest)
  • Windows 10 SDK 10.0.18362.0
  • USB Device Connectivity (optional)
  • C++ UWP Support for v142 Buildtools (ARM64)

• Git installed, check the Option to add it to the path during installation

• Nuget command line downloaded and added to your path

• At least 40 GB of free disk space

Part 1: Building the MRSV-Unity Package

Before starting with the building process, it is extremely important that you have the exact software that is described here installed on your system. If you don't use the exact same version, the build might fail

Step 1: Downloading & Setup all files

To download all the neccessary files, we first need to specify the folders in which we want the files to be downloaded into. To avoid conflicts during build that can arise due to exceeding the max path length limit, you should create a folder that is very close to the root of your drive. For example, I created a folder C:\Dev\Software and a folder C:\Dev\Tools.

I put the nuget.exe into the C:\Dev\Tools folder and added it to the path. Now open a command line window with administrator rights and navigate to your path:

cd C:\Dev\Software

Now we'll clone this repository as a submodule. Type the following command into the command line and wait until git has finished:

git init git submodule add https://github.com/microsoft/MixedReality-SpectatorView

We should have a copy of this repo in our folder now. Now we execute a .bat file inside of the repo, that will download, install and build the Unity Package...at least in theory. The process will fail a at first, but we'll fix that later. Nevertheless, it is still very important that you execute this command once, so that all files are being downloaded. While doing this, pay very close attention the the console output! Luckily, the logging is quite good and will remind of any errors that exist.

Let's start by executing MixedReality-SpectatorView\tools\scripts\CreateUnityPackage.bat -HardCopySymbolicLinks

Depeding on your internet connection and CPU, this might take from ten minutes to a few hours. A lot of files will be downloaded now and a build attempt will be made.

When the process is done, you should see something similar to this:

Notice all the 'FALSE' flags on the build results, which indicate an unsuccessfull build. Let's find the culprit!

2: Fixing TIFF UWP

When we scroll up on our log we see this error:

This is caused by an incompatibility of the TIFF library with the UWP-Platform. To fix this, we need to patch the source code file. Luckily, vcpkg (the library manager used in this repo) allows us to patch faulty files.

To do this, we'll go into the Tiff-Lib Source folder with the command line via this line ( Replace vX.X.X-xxxxxx.clean with whatever your folder is called there):

cd MixedReality-SpectatorView\external\vcpkg\buildtrees\tiff\src\vX.X.X-xxxxxxxxx.clean

Then we initiate an empty git repository, which allows us the create the patch file, add the current version of the files to it and commit it:

git init . git add . git commit -m "temp"

Now we can modifiy the files. To do that, we go into the MixedReality-SpectatorView\external\vcpkg\buildtrees\tiff\src\vX.X.X-xxxxxxxxx.clean\libtiff\ folder and open tif_win32.c inside of a text editor

Now we need to delete this code at around line 280:

fd = (thandle_t)CreateFileA(name, (m == O_RDONLY)?GENERIC_READ:(GENERIC_READ | GENERIC_WRITE), FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, dwMode, (m == O_RDONLY)?FILE_ATTRIBUTE_READONLY:FILE_ATTRIBUTE_NORMAL, NULL);

And replace it with this code:

`#if (_WIN32_WINNT >= _WIN32_WINNT_WIN8) fd = (thandle_t)CreateFile2(name, (m == O_RDONLY)?GENERIC_READ:(GENERIC_READ | GENERIC_WRITE), FILE_SHARE_READ | FILE_SHARE_WRITE, dwMode, NULL);

else

fd = (thandle_t)CreateFileA(name,
    (m == O_RDONLY)?GENERIC_READ:(GENERIC_READ|GENERIC_WRITE),
    FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, dwMode,
    (m == O_RDONLY)?FILE_ATTRIBUTE_READONLY:FILE_ATTRIBUTE_NORMAL,
    NULL);

endif`

And the same for this code at around line 330, delete:

fd = (thandle_t)CreateFileW(name, (m == O_RDONLY)?GENERIC_READ:(GENERIC_READ|GENERIC_WRITE), FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, dwMode, (m == O_RDONLY)?FILE_ATTRIBUTE_READONLY:FILE_ATTRIBUTE_NORMAL, NULL);

replace it with:

`#if (_WIN32_WINNT >= _WIN32_WINNT_WIN8) fd = (thandle_t)CreateFile2(name, (m == O_RDONLY)?GENERIC_READ:(GENERIC_READ | GENERIC_WRITE), FILE_SHARE_READ | FILE_SHARE_WRITE, dwMode, NULL);

else

fd = (thandle_t)CreateFileW(name,
    (m == O_RDONLY)?GENERIC_READ:(GENERIC_READ|GENERIC_WRITE),
    FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, dwMode,
    (m == O_RDONLY)?FILE_ATTRIBUTE_READONLY:FILE_ATTRIBUTE_NORMAL,
    NULL);

endif`

Now save the file and close it.

Back in our command line, we get the difference between our last commit and the changed file, and output it as a diff file. This is our Patch file. To do this, we enter:

git diff --ignore-space-at-eol --output Fix_UWP_Tiff.patch

Now we should have a Fix_UWP_Tiff.patch file in our vX.X.X-xxxxxxxxx.clean folder. Move this file to MixedReality-SpectatorView\external\vcpkg\ports\tiff\

To apply the patch, we open the "portfile.cmake" in the same folder, and add Fix_UWP_Tiff.patch after "FindCMath.patch" on line 66, save it.

Now we can go back to the parent folder of the MRSV-Repo with cd C:\Dev\Software\ and then try to build again MixedReality-SpectatorView\tools\scripts\CreateUnityPackage.bat -HardCopySymbolicLinks

Now after the build has concluded, most build results should be "TRUE" (Exept for the optional stuff like Elgato, Blackmagic ect.)

3: Fixing the QR-Code dependencies

In the current state of the project, important C-Sharp files for the QR-Code detection are simply removed from the package. Without them however, the QR-Code detection won't work. To fix this problem, open MixedReality-SpectatorView\tools\Scripts\ExternalDependencyHelpers-psm1 inside of a text editor, and remove these lines, then save it:

# TODO - remove this deletion step once qr code nuget packages don't break mac builds Write-Host "Removing c# files that break in Unity packages for QRCode Dependencies in directory $contentFolder\*.cs*" Remove-Item -Recurse $contentFolder -Include *.cs*

Rebuild the package again with the command MixedReality-SpectatorView\tools\scripts\CreateUnityPackage.bat -HardCopySymbolicLinks

4: Copy QR-Code dependencies into the package

The package has been built, but they are missing the QR-Code dependencies. So we need to copy the contents of the folder "MixedReality-SpectatorView\external\MixedReality-QRCodePlugin\UnityFiles" into the folder "MixedReality-SpectatorView\packages\com.microsoft.mixedreality.spectatorview.1.2.0\SpectatorView.ExternalDependencies\UnityFiles"

Done! The package is now ready to be used inside of Unity 👍

Part 2: Building the Hololens 2 & Android Apps

For this tutorial, I'll be using the sample scenes that are shipped with this repository. If you skipped Part 1 and just downloaded the package, you still need to clone this git repository, to use the Sample scenes in there. Clone the repository somewhere close to the root of your C:\ drive (For example C:\Dev\Software).

Step 1: Configuring the project

Open up the "C:\Dev\Software\MixedReality-SpectatorView\samples\SpectatorView.Example.Unity" folder with Unity 2019.2.21. If asked about upgrading/reimporting the project due to a newer Unity version, go ahead. Ignore the errors in the console for now.

Now we'll add the MRSV-Package via Window > Package Manager > + > Add Package from Disk Select the package.json from our package (C:\Dev\Software\MixedReality-SpectatorView\packages\com.microsoft.mixedreality.spectatorview.1.2.0\package.json)

There are two remaining errors in the console, you can ignore them for now.

Now we open our Sample Scene (Assets > Scenes > SpectatorView.Hololens) and Update the Asset Caches (Spectator View > Update All Assets Caches)

Step 2: Setting the right Windows SDK in the registery

MRSV only works with SDK 10.0.18362.0. Later SDKs have a bug in the Networking sockets. You can theoretically choose which SDK Unity should use in the build dialog, but this doesn't actually work. Instead we need to do this in the windows register:

Open the windows search bar and type "Registry", open the Registry-Editor. Open the folder:

Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Microsoft SDKs\Windows\v10.0

In there, open the Entry "ProductVersion" and change it's value to "10.0.18362.0". Done!

Step 3: Building the Hololens App

In the SpectatorView.Hololens Scene, select the "SpectatorView" GameObject and click on "Hololens" on the Platform switcher. This should configure the project correctly for the Hololens. Also make sure that the "Role" is set to "User" on the Spectator View Script. Now, open the build settings, and make sure they look like this:

You can disable "Development Build", "Script Debugging" and "Wait For Managed Debugger", but I'd recommend it, so that you can notice any errors that might have.

Hit Build, and build the project to any directory you want.

When the build is done, open the newly created Visual Studio solution inside your build folder with Visual Studio 2019. Power on your Hololens, and connect your Hololens to the same network as your computer.

Check that The "SpectatorViewExampleUnity" Projekt is set as startup project (Bold text)

Now, configure your project as Release | ARM64 | Remote Machine

Go to the Debug properties:

Go to Configuration Properties > Debugging > Machine Name and enter the IP of your Hololens

Now hit the "Play/Build" button, and your app will be uploaded to the Hololens.

Step 4: Building for Android In the SpectatorView.Hololens Scene, select the "SpectatorView" GameObject and click on "Android" on the Platform switcher. Also make sure that the "Role" is set to "Spectator" on the Spectator View Script. Now, open the build settings, and make sure they look like this:

"Development Build" and "Script Debugging" is optional again, but is helpful for debugging if anything goes wrong. You can use Android Logcat to watch for errors.

Build it! If asked about a gradle conflict, select "No". Install it on your Android device with ADB

Step 5: Running the Apps

Connect both of your devices to the same network. You may have to disable mobile data on your phone. Start both apps. When you can see the three boxes on your Hololens, enter the IP adress of your hololens in the Android App and connect. When a QR-Code shows up, grant permissions to use the camera inside of the Hololens and hold to it's front camera to scan it. You may have to restart the Hololens 2 and Android app after granting permissions.

The scene should now hopefully also show on your android device! Good Luck!

[paulio]

FYI I believe this is the patch you're referring to for the Networking problem so you don't have to hard wire the Windows 10 SDK; https://forum.unity.com/threads/il2cpp-failing-in-windows-machine.891436/#post-5944052

[mallikhrao]

I tried rebuilding with above modifications, but still failing at the Tiff issue. Also can you provide alternative link to https://github.com/ExperimentalSurgery/MRSPV-Unity-Package. It says package not found

[ChristopherRemde]

Sorry, the repo was still set to private, it should work now!

Can you go to: MixedReality-SpectatorView\external\vcpkg\buildtrees\tiff\src\vX.X.X-xxxxxxxxxx.clean\libtiff\ and open tif_win32.c and take a look if the patch has been applied correctly? You should see the code you pasted in step 2 in there. If not, double check that you followed the steps correctly and delete all folders from tiff/src/, then try to rebuild again.

[mallikhrao]

Thank you Christopher, seems like it is working now. I just checked the build and it succeed. Appreciate your quick response.

[AbhiKana]

Hi there, I tried Building with the above steps and am getting the following error

I also tried building with -NoDownloads but still got the same error.

[ChristopherRemde]

Hey! Seems like something from the patching step went wrong. Triple check and repeat every step from "2: Fixing TIFF UWP", maybe also try deleting the files you created first. It's super easy to get something wrong in this step, so make sure you follow the instructions exactly!

[AbhiKana]

Hi I also got to confirm we don't want to commit the empty git repo that we create after making the changes to tif_win32.c.

Also when I did the steps of adding the Fix_UWP_Tiff.patch in portfile.cmake

and when I run the createunitypackage script it creates a .tmp folder which has libtiff-v4.4.0

[ChristopherRemde]

Yes that's right, you don't want to commit the git repo after making the changes.

After you made sure that you executed all steps correctly, try to delete these folders and then run the script again!

[Isometric-Bacon]

Fantastic guide. Thankyou for taking the effort to type this out. This is one of the most bizarrely packaged unity projects i've ever come across.

With your help, I managed to get the project deployed and can run my app in spectator view in Android and on the Hololens.

Sadly the Android app lacks alot to be desired, stuck in portrait mode, with a fixed focus. Did anyone have any luck with the SLR option?

I am trying now to get an Elgato HD60 S+ to work, but the compositor just gives me a green screen, despite it working in the 4k capture software and windows camera app. I haven't done the TIFF fix, but I suspect that would be unrelated.

:Edit: Now I think it's because the HD60 S+ use UVC usb - looking at the git it looks like the older versions of this card, confusingly named similarily use Direct Show.

Ah well. Lessons learned. Guess I'm gonna stick to android for the time being.

[Isometric-Bacon]

Ah at my wit's end with this project. After getting everything running fine, I migrate PCs and cannot get it working again, despite everything being the same.

This time it appears that ARCore is not installing correctly, in that Google Play services pops up and notifies me that my phone needs the latest version of Google AR services when I run the built app. It doesn't in my earlier (identical repo) builds.

[ChristopherRemde]

Just to be sure, the old APK you build does still work, just the new one not?

But yeah this project is a big mystery, after working fine for several months the app on my phone just crashes, even after reinstalling, and not changing anything on my phone. Haven't come around to debug it yet

[Isometric-Bacon]

Ah sorry to hear that.

Yep, my old APK is fine.

I suspect it's something to do with the AR core package failing to install. I also had integrated MRTK 2.8 and had updated Unity to 2.4.22 like some of the other posts mention here. I recall I had to update ARCore / foundation packages to 3.1.3 to have it be compatible.

Sadly doing this again nothing works and I'm sure it's something esoteric I've missed, (or my repo is set up wrong).

Pity MS aren't still supporting this - it's such a powerful use case for showcasing this nascent technology.

[ChristopherRemde]

Agreed, it's probably one of the most important tools to get people to understand how AR works. Sadly, Microsoft doesn't treat it this way.

If you can find a fix for your bug let us know, maybe it'll help someone else!

[dudziakl]

Thank you @ChristopherRemde for your great guide! I did everything as you said and was able to fix Tiff library and build both for HoloLens and Android. It is very important to stay on Unity 2019.2.21. I tried to update the project to 2019.4.34 but the android app doesn't work. I had the same result as @Isometric-Bacon where Google Play message appears and requires upgrading AR Core to the latest version. It doesn't work as I have the latest version. Reinstalling the 2019.2x version resolved the problem and I was able to connect successfully. The hybrid scenario where HoloLens has 2019.4.x and phone has an older version, doesn't work either.

On the HoloLens log file looks like that

QR Code Added: sv1 QRCode location unknown or not yet available. <- this is probably the issue MarkerDetector: Markers Updated MarkerDetector: Processed all markers MarkerDetector: Marker with id: 1 wasn't found yet. <- this is going in the loop

and the mobile app gets stuck at QR code screen. I will let you know if I found a solution.

I'm curious if anyone was able to upgrade this project to a less ancient Unity version.

[ChristopherRemde]

Thanks for your investigation! I'll highlight the part that you need to stay on this specific Unity Version in the Title post. I also tried with other versions from Unity, and didn't succeed at all. I actually had to downgrade our project to match the 2019.2.21 version