SpectatorView: Menu Items Not Appearing

[HoloNoob]

I am having issues getting the Spectator View and HoloToolKit menu items to appear on the menu bar.

I am following the instructions here: https://github.com/Microsoft/HoloLensCompanionKit/blob/master/SpectatorView/Compositor/README.md

and have got as far as 'Load your Unity project' but I don't have the menu item to continue. The HoloToolKit menu item did appear but has now disappeared.

Any ideas on what I have missed?

Thanks in advance

[fieldsJacksonG]

Does your project use the HoloToolkit: https://github.com/Microsoft/HoloToolkit-Unity

You will need to add this to your project.

Then you will need to import the SpectatorView unity package: https://github.com/Microsoft/HoloLensCompanionKit/tree/master/SpectatorView/UnityPackage

And copy the DLLs you built from the compositor sln to your project using the CopyDLL.cmd script with the path to your Assets directory.

The project is currently being updated with a unity package for a min-set of the 1.5.5.0 toolkit release which should make this process easier.

[fieldsJacksonG]

Also: do you have any error messages in your Unity console?

Any errors in this console will prevent scripts from loading until they are fixed.

[fieldsJacksonG]

Per my comment above, we have added additional guidance for using spectator view with the 1.5.5.0 drop of the HoloToolkit.

[HoloNoob]

Thanks. I started again with the 1.5.5.0 versions you uploaded last night and that resolved the menu item issue. However I do have errors in Unity. See image:

If I then press play in the Unity window I get 999+ DLLNotFoundExceptions. I assume the CopyDLL.cmd is directing the Compositor sln DLL changes to the correct location?

I have added the directory for the project assets folder as below:

` Setlocal EnableDelayedExpansion

set /a returnValue=0

if "%~1" NEQ "" ( call :CopyDLL "%~1" )

REM Copy DLLs to Sample Unity projects. call :CopyDLL "%~dp0\Samples\HL SV 5\Assets\"

if %returnValue% NEQ 0 (`

[fieldsJacksonG]

Did you build the dlls for release\x64 and release\x86? Make sure you did not build these for Arm. Note also that when working off of the Unity packages, you will have to edit the meta files or copy the meta files from the sample project, per the TODO files in the Plugins directory. Edit: if you pull the latest changes, the CopyDLL script will copy the correct meta files to your project.

When you run CopyDLL, what is your output?

What is in your assets\addons\holographicCameraRig\Plugins directory?

What frame provider are you using? Do you see color frames in the Calibration sln? Do you have the correct files downloaded for your frame provider and referenced in dependencies.props?

[fieldsJacksonG]

Also: you do not need to edit the CopyDLL script, you can call it from a cmd prompt like this:

CopyDLL.cmd "%path to your assets directory%"

[HoloNoob]

DLLs were built for x64 and x86, not ARM.

---

I have replaced my edited version of the CopyDLL.cmd with the latest version as advised and run this from a cmd prompt (I was double-clicking previously!).

Have also added PluginMetaFiles to the Compositor folder.

Subsequent DLL output is:

> 

C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView>CopyDLL.cmd C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\Samples\HL SV 5\Assets

===================================================
Copy DLL's to your Unity projects.
Include an optional path to the Assets path of your Unity project to copy DLLs to that project.
Otherwise, just copy to the included sample projects.
===================================================

===================================================
Copy DLL's to C:\Unity

"C:\Unity\Addons\HolographicCameraRig" does not exist.
===================================================

===================================================
Copy DLL's to C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Samples\SharedHolograms\Assets\

Copy UnityCompositorInterface files:
        1 file(s) copied.
        1 file(s) copied.
        1 file(s) copied.
        1 file(s) copied.

---------------------------------------------------
Copy SpatialPerceptionHelper files:
        1 file(s) copied.
        1 file(s) copied.
        1 file(s) copied.
        1 file(s) copied.

---------------------------------------------------
Copy CompositorDLL:
        1 file(s) copied.
        1 file(s) copied.
        1 file(s) copied.
        1 file(s) copied.

---------------------------------------------------
Copy Canon SDK:
        1 file(s) copied.
        1 file(s) copied.
        1 file(s) copied.
        1 file(s) copied.

---------------------------------------------------
Copy OpenCV:
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\x64\Release\opencv_ffmpeg310_64.dll
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\x64\Release\opencv_ffmpeg320_64.dll
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\x64\Release\opencv_world310.dll
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\x64\Release\opencv_world310d.dll
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\x64\Release\opencv_world320.dll
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\x64\Release\opencv_world320d.dll
        6 file(s) copied.

---------------------------------------------------
Copy GitIgnore:
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\.gitignore
        1 file(s) copied.

---------------------------------------------------
Copy meta files:
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\WSA\x64\SpatialPerceptionHelper.dll.meta
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\WSA\x64\SpatialPerceptionHelper.pdb.meta
        2 file(s) copied.
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\WSA\x86\SpatialPerceptionHelper.dll.meta
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\WSA\x86\SpatialPerceptionHelper.pdb.meta
        2 file(s) copied.
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\x64\CompositorDLL.dll.meta
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\x64\CompositorDLL.pdb.meta
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\x64\EDSDK.dll.meta
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\x64\EdsImage.dll.meta
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\x64\opencv_ffmpeg310_64.dll.meta
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\x64\opencv_world310.dll.meta
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\x64\opencv_world310d.dll.meta
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\x64\UnityCompositorInterface.dll.meta
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\x64\UnityCompositorInterface.pdb.meta
        9 file(s) copied.
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\x86\CompositorDLL.dll.meta
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\x86\CompositorDLL.pdb.meta
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\x86\EDSDK.dll.meta
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\x86\EdsImage.dll.meta
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\x86\UnityCompositorInterface.dll.meta
C:\Unity Projects\HoloLensCompanionKit-master\SpectatorView\\Compositor\PluginMetaFiles\x86\UnityCompositorInterface.pdb.meta
        6 file(s) copied.
===================================================

***************************************************
***************************************************

   ERROR: One or more files did not copy.
          Please review log.

***************************************************
***************************************************

---

I am still getting these errors in Unity:

> Failed to load 'Assets/Addons/HolographicCameraRig/Plugins/x64/UnityCompositorInterface.dll' with error '%1 is not a valid Win32 application.
'.
^ this error x3

> DllNotFoundException: UnityCompositorInterface
CompositorWindow..cctor () (at Assets/Addons/HolographicCameraRig/Editor/CompositorWindow.cs:75)
Rethrow as TypeInitializationException: An exception was thrown by the type initializer for CompositorWindow

---

I am using a Blackmagic Decklink SDI 4K and have the below folder referenced in the dependencies.props:

<DeckLink_inc>C:\Blackmagic DeckLink SDK 10.8.3\Win\include</DeckLink_inc>

`// FrameProvider type - Exactly 1 of these should be true:
//TODO: Set this to true if using a BlackMagic DeckLink capture card.
#define USE_DECKLINK   TRUE
//TODO: Set this to true if using an Elgato capture card.
#define USE_ELGATO     FALSE
//TODO: Set this to true if using OpenCV to get frames from a camera or capture card.
#define USE_OPENCV     FALSE`

---

Contents of 'assets\addons\holographicCameraRig\Plugins directory' as in image:

Thanks again for your help so far!

[fieldsJacksonG]

Two things jump out at me to try:

1. when you call copydll, ensure your file path has quotes around it if there are spaces. Looks like it's failing to copy because C:\Unity does not have the spectator view add on. You do have one of the copied files in your Plugins directory, likely from when you edited the copydll script earlier.

2. I see you are using the CanonSDK. I think this might be the issue: I had a bug that I fixed in a checkin yesterday where the same Canon architecture was being copied to x86 and x64. If you pull the Compositor changes and rebuild (not just build - you can do a build clean first to guarantee this) you will get the correct architectures. Also ensure that your dependencies.Props file shows the Canon sdk as: c:\CanonSDK\Windows and not c:\CanonSDK\Windows\EDSDK_64

(Please also note: when using the Canon SDK, a >4K camera will be added to your spectator view prefab to add holograms to your Canon capture. This will likely slow down your frame rate to a point not suitable for live demonstration or video capture, so you should only be using the USE_CANON_SDK flag if targeting a photo shoot. When doing a photo shoot, you will need to attach a mini USB cable from your PC to your Canon camera as well.)

[HoloNoob]

Thanks for the above. That removed the errors from the CopyDLL output and allowed SpectatorView/Compositor window to appear in Unity. I have also amended the CanonSDK dependecies.props directory.

However I now get these two errors in Unity:

1. this error when opening Compositor in Unity:

Failed to call static function Reset because an object was provided UnityEditor.EditorWindow:GetWindow(Type, Boolean, String, Boolean) CompositorWindow:ShowWindow() (at Assets/Addons/HolographicCameraRig/Editor/CompositorWindow.cs:108)

2. this when pressing play in Unity:

Sharing Service IP must be populated. UnityEngine.Debug:LogError(Object) SpectatorView.SpectatorViewManager:Awake() (at Assets/Addons/HolographicCameraRig/Scripts/SpectatorViewManager.cs:53)

[fieldsJacksonG]

https://github.com/Microsoft/HoloLensCompanionKit/blob/master/SpectatorView/Samples/README.md#setup

Error 1 is benign and just happens sometimes.

Error 2 is because you are not populating the IPs in your SpectatorViewManager prefab. Click on SpectatorViewManager in your hierarchy, and look at it in the Inspector. You will see a few IPs:

1. "Spectator View IP" - populate this with the IP of the HoloLens on your spectator view rig.
2. "Sharing Service IP" - populate this with your PCs local IP - this is the IP that the sharing service is running on. You will need to launch the sharing service from HoloToolkit\Sharing Service\Launch Sharing Service.
3. "Client HoloLens CSV' - populate this with a comma separated list of all of the other HoloLens devices in your experience. These will be used to deploy to if you are using the "App Management" build and deploy buttons below these IP fields.
4. "Local Computer IP" - Only necessary if you have more than 1 spectator view rig. This is to disambiguate which PC your spectator view rig is connecting to.

[HoloNoob]

Thanks for the above. IPs now entered into SpectatorViewManager but still no dice.

I have also updated the IP address in the stdafx.h as below and turned off SSL via device portal. Hololens user and password also updated.

Now I see this error when Updating IP in SpectatorViewManager:

NullReferenceException: Object reference not set to an instance of an object SpectatorView.SpectatorViewManager.UpdateSpectatorViewIP () (at Assets/Addons/HolographicCameraRig/Scripts/SpectatorViewManager.cs:112) SpectatorView.SpectatorViewManagerEditor.OnInspectorGUI () (at Assets/Addons/HolographicCameraRig/Editor/SpectatorViewManagerEditor.cs:62) UnityEditor.InspectorWindow.DrawEditor (UnityEditor.Editor editor, Int32 editorIndex, Boolean rebuildOptimizedGUIBlock, System.Boolean& showImportedObjectBarNext, UnityEngine.Rect& importedObjectBarRect) (at C:/buildslave/unity/build/Editor/Mono/Inspector/InspectorWindow.cs:1239) UnityEditor.DockArea:OnGUI()

And this output when launching sharing service:

Thanks again for your help.

[fieldsJacksonG]

Did you press 'Play' in Unity after launching the Sharing Service? After pressing 'Play' you should see text in the Sharing Service: "User %username% at address %ip% has joined session Default"

Did you allow your sharing service through your firewall?

Did you set your Sharing Service IP as 192.168.2.127

Make sure you don't have any spaces in your Sharing Service IP.

Your Unity project's hierarchy should at a minimum have these prefabs after pressing 'Play':

The NullReferenceException implies to me that the HolographicCameraManager was not created. Were you editing the SpectatorViewManager in your Hierarchy window? (Not the one in the Project window)

Also ensure that your SpectatorViewManager has valid prefabs in the "Holographic Camera Manager" "Anchor" and "Sharing" prefabs.

[fieldsJacksonG]

If you try with the sample project, do you see the same results?

[HoloNoob]

Thanks, I have checked all the above and they are all correct. I am no longer getting error messages. Sharing session successfully joined

I am only getting a strobing or static green feed in the compositor window though. Canon camera is connected via HDMI to PC (additional USB connection is only requiring for live streaming correct?) and Hololens is on same network as PC. What am I missing?

[HoloNoob]

Same outcome with Sample project also.

[fieldsJacksonG]

You're almost there. Make sure your camera is plugged into the HDMI-in port on your capture card.

Also make sure your camera is setup to output frames over HDMI: https://github.com/Microsoft/HoloLensCompanionKit/blob/master/SpectatorView/README.md#camera-setup

A USB connection is only necessary if you want to use the Canon SDK. You would use a second HDMI from the HDMI-out port if you wanted to live stream.

[fieldsJacksonG]

You can also use the Blackmagic media express application to sanity check that you are getting color frames from your camera. This is installed when you install the Blackmagic runtime.

(Open that application and select the "playback" tab.)

Note: When you are using spectator view or the media express application, you will need to make sure no other applications are accessing your capture card.

Note 2: The version of the Blackmagic SDK you downloaded should match the version of the runtime you installed.

[HoloNoob]

Excellent! And yes schoolboy error, the HDMI wasn't in the capture card, which only has SDI inputs. Just waiting for a HDMI to SDI convertor to arrive and then hopefully we'll be even closer.

Thanks and have a great weekend.

[fieldsJacksonG]

That would do it - note the flashing is the Decklink device checking whether the signal is yuv or rgba (a null frame is black in rgba and green in uyvy)

Closing this issue since you are now seeing menu items.