Deployment on HoloLens fails

[IoannisKaragiannis]

Description

When I try to deploy my MLAPI example on my HoloLens device I get the following errors:

2> Copying unprocessed assemblies... 2> Running AssemblyConverter... 2> Failed to fix references for method System.Byte[] MLAPI.NetworkingManagerComponents.Cryptography.CryptographyHelper::Decrypt(System.Byte[],System.Byte[]) 2> Failed to fix references for type MLAPI.NetworkingManagerComponents.Cryptography.CryptographyHelper 2> System.Exception: Failed to resolve System.Security.Cryptography.RijndaelManaged 2> at Unity.ModuleContext.Retarget(TypeReference type, GenericContext context) 2> at Unity.FixReferencesStep.Visit(MethodDefinition method, GenericContext context) 2> at Unity.FixReferencesStep.Visit(TypeDefinition type) 2> at Unity.TypeDefinitionDispatcher.DispatchType(TypeDefinition type) 2> at Unity.TypeDefinitionDispatcher..ctor(ModuleDefinition module, ITypeDefinitionVisitor visitor) 2> at Unity.FixReferencesStep.ProcessModule() 2> at Unity.ModuleStep.Execute() 2> at Unity.FixReferencesStep.Execute() 2> at Unity.Step.Execute(OperationContext operationContext, IStepContext previousStepContext) 2> at Unity.Operation.Execute() 2> at Unity.Program.Main(String[] args)

Do you have any ideas what's causing this failure and how can I fix it?

Thanks

Your Environment

• Unity Version: 2017.4.0f1
• MLAPI Version: v1.1.1

[TwoTenPvP]

Seems like RijndaelManaged doesn't exist in the runtime you are targeting. I'll see what I can do. I doubt Unity will implement this even if they got a bug request on it.

[IoannisKaragiannis]

Oh I see. Is there any chance I can bypass this for now?

[TwoTenPvP]

Add

return new byte[0];

At the top of the Decrypt and Encrypt methods in CryptographyHelper and then comment out the "non reachable" method body that is left. Might also have to remove a the "using S.S.Cryptography"

[IoannisKaragiannis]

I will try that and let you know. Thanks.

[GabrielTofvesson]

If necessary, I could add a custom Rijndael implementation if RijndaelManaged couldn't be found

[TwoTenPvP]

Could do, Mono has the RijndaelManaged, but when targeting HoloLens I think you have to run UWP which targets the C++ windows API.

Edit: https://docs.unity3d.com/Manual/windowsstore-plugins.html Seems like my assumption was correct. We need conditional compile to depending on the platform which is not possible since we are a class lib and already post-firstpass compile.

A custom Rijndael implementation would be great @GabrielTofvesson.

[IoannisKaragiannis]

I recompiled MLAPI with the recommended changes in the CryptographyHelper but I now get this:

2> Copying unprocessed assemblies... 2> Running AssemblyConverter... 2> Failed to fix references for field System.Security.Cryptography.RNGCryptoServiceProvider MLAPI.NetworkingManagerComponents.Cryptography.EllipticDiffieHellman::rand 2> Failed to fix references for type MLAPI.NetworkingManagerComponents.Cryptography.EllipticDiffieHellman 2> System.Exception: Failed to resolve System.Security.Cryptography.RNGCryptoServiceProvider 2> at Unity.ModuleContext.Retarget(TypeReference type, GenericContext context) 2> at Unity.FixReferencesStep.Visit(FieldDefinition field, GenericContext context) 2> at Unity.FixReferencesStep.Visit(TypeDefinition type) 2> at Unity.TypeDefinitionDispatcher.DispatchType(TypeDefinition type) 2> at Unity.TypeDefinitionDispatcher..ctor(ModuleDefinition module, ITypeDefinitionVisitor visitor) 2> at Unity.FixReferencesStep.ProcessModule() 2> at Unity.ModuleStep.Execute() 2> at Unity.FixReferencesStep.Execute() 2> at Unity.Step.Execute(OperationContext operationContext, IStepContext previousStepContext) 2> at Unity.Operation.Execute() 2> at Unity.Program.Main(String[] args)

So it's in the DiffieHellman now. Any recommendation?

[TwoTenPvP]

Ugh, seems like we are not allowed any of the Cryptography namespaces. I'm unsure how to solve this permanently. A temporary fix is as easy as removing the use of the Cryptography, but that would not allow you to use Encryption.

[IoannisKaragiannis]

Does it have anything to do with my Configuration on the Player? This is how I have it now

[TwoTenPvP]

Don't think so. Just remove all cryptography stuff and recompile. (You only have to remove usage of the System.Security.Cryptography namespace classes) replace outputs etc with dummy values)

[IoannisKaragiannis]

But in general as you said this will be a temporary solution. Do you think we could come up with a more permanent one?

[TwoTenPvP]

That's what I will be working on. I have no clue how to solve it atm.

[IoannisKaragiannis]

In Missing .NET APIs in Unity and UWP they state that some of the System.Security^* namespaces (e.g.: System.Security.Cryptography.X509Certificates,) are not available when you build a Unity game for UWP. They instead recommend to use Windows.Security^*. I found this useful. What do you think?

[TwoTenPvP]

https://github.com/TwoTenPvP/MLAPI/commit/f41947112666f63d2eab7fd25412dc3b0d0f86f6 I added a pre-processor directive to disable cryptography for now.

But thanks for posting that resource! While that helps to solve the "real" problem. We are still a class lib and we are post first pass compilation :/. This means that we are unable to do a simple:

#if NETFX_CORE
//Windows API
#else
//.NET API
#endif

Once again, a code weaver is a solution I am looking at, especially for the SyncedVar system to get away from the Reflection and nesting hell we have right now (Though it's gotten fairly optimized with resent caching applied in v1.2.2+).

We can't really evaluate the platform we run on at runtime and then choose what method to use due to the fact that Unity themselves weave and even transpile the code in some cases, from IL to C++.

Still open for suggestions. But as long as the Cryptography stuff exists in the Windows API @GabrielTofvesson doesn't have to supply us with a Rijndael implementation.

The last alternative would be to have our own implementations of AES / Rijndael and a cryptographic random number generator.

[IoannisKaragiannis]

Alright. I will test it tomorrow. I'm not quite sure if the Rijndael implementation exists for the Windows API. There are lots of Cryptography stuff, but I don't think they have all the features that System.Security.Cryptography has. Thanks again.

[TwoTenPvP]

Well AES is one of the most basic things, very likely to exist. Don't have time to look into it right now. For the moment, use the DISABLE_CRYPTOGRAPHY directive when compiling.

[IoannisKaragiannis]

I still get some errors for the networking manager. It's in the OnValidate function. You are using cryptographic stuff in there. Moreover you are using the System.Security.Cryptography, and MLAPI.NetworkingManagerComponents.Cryptography namespaces. I will introduce the

#if !DISABLE_CRYPTOGRAPHY
// normal code
#endif

logic where required.

[TwoTenPvP]

Do the namespaces themselves prevent compilation?

[IoannisKaragiannis]

Nope. I just commented them out as well. Now it seems to be working. That's what I did

            if (!NetworkConfig.EnableEncryption)
            {
                RegenerateRSAKeys = false;
            }
            else
            {
                if(RegenerateRSAKeys)
                {
#if !DISABLE_CRYPTOGRAPHY
                    using (RSACryptoServiceProvider rsa = new RSACryptoServiceProvider())
                    {
                        rsa.PersistKeyInCsp = false;
                        NetworkConfig.RSAPrivateKey = rsa.ToXmlString(true);
                        NetworkConfig.RSAPublicKey = rsa.ToXmlString(false);
                    }
#endif
                    RegenerateRSAKeys = false;
                }
            }

[TwoTenPvP]

https://github.com/TwoTenPvP/MLAPI/commit/a44f939e8f1a155c388fe9a88715322afda3c86f should be fixed?

[IoannisKaragiannis]

Could you please have a release with the DISABLE_CRYPTOGRAPHY option?

[TwoTenPvP]

Probably not. This requires me to compile twice every time I want to release, update the CI to build twice and update the installer. I will most likely look for a more permanent solution to this problem.

[IoannisKaragiannis]

No worries. I want to inform you about something else. So far, I have been mainly working with your v1.1.1 release. It was today that I downloaded your master and I had some issues with my multiplayer game. I have 3 machines. My desktop (Windows 10) hosts the Server. The two laptops (Windows 7) operate as Clients. With v1.1.1 everything was performing smoothly. This is not the case any more with the master I downloaded today.

1) I start the Server on my Desktop. The Server only has a scene with one Networked Object, a cube in this case.

2) Client-A joins the game. I can see the player-bot of Client-A and the cube both on the Server and on the Client-A scene. And they are perfectly synced.

3) Client-B joins the game. The player-bot of Client-B appears on the Server and on Client-B scene but not on Client-A scene. Client-A can see the cube which is moved by the player-bot of Client-B, but not the player-bot itself. It's like a ghost is moving the cube. So, the player-bot of Client-B is not spawned in the scene of Client-A.

You must have introduced a bug in one of your latest releases, but since I have not been following them, I can't say where the problem is. I can start working with v1.2.0 and see if there is any problem there. It's good to figure out which is the most recent stable release, and try to detect the bug from that release and on. Do you by any chance have any idea what could have gone wrong?

Thanks.

[TwoTenPvP]

I have no clue? If spawn behavior has broken in a recent release, please open a new issue describing the bug and please try to pinpoint the release (or even better, the commit) where the bug was introduced since this is what I will probably do anyways.

I will try to find time to look into this issue and the bug you are speaking about now. I have so far been busy reworking the SyncedVar system and creating the new NetworkProfiler.

[IoannisKaragiannis]

Yes I saw what you are working on. Alright, I will open a new issue. Thanks.

[IoannisKaragiannis]

Today I tried the following.

I downloaded the v1.1.1 version, I compiled it, and I inserted in my project's Asset folder the MLAPI.dll, MLAPI.xml, IntXLib.dll, IntXLib.xml.

When I try to deploy it on HoloLens I get the following error:

1> Copying unprocessed assemblies... 1> Running AssemblyConverter... 1> Failed to fix references for interface System.Runtime.Serialization.ISerializable 1> Failed to fix references for type IntXLib.FhtMultiplicationException 1> System.Exception: Failed to resolve System.Runtime.Serialization.ISerializable 1> at Unity.ModuleContext.Retarget(TypeReference type, GenericContext context) 1> at Unity.FixReferencesStep.Visit(InterfaceImplementation interfaceImplementation, GenericContext context) 1> at Unity.FixReferencesStep.Visit(TypeDefinition type) 1> at Unity.TypeDefinitionDispatcher.DispatchType(TypeDefinition type) 1> at Unity.TypeDefinitionDispatcher..ctor(ModuleDefinition module, ITypeDefinitionVisitor visitor) 1> at Unity.FixReferencesStep.ProcessModule() 1> at Unity.ModuleStep.Execute() 1> at Unity.FixReferencesStep.Execute() 1> at Unity.Step.Execute(OperationContext operationContext, IStepContext previousStepContext) 1> at Unity.Operation.Execute() 1> at Unity.Program.Main(String[] args)

I also realized that the build on Unity comes with some errors like the following:

Reference rewriter: Error: type System.Runtime.Serialization.ISerializable doesn't exist in target framework. It is referenced from IntXLib.dll at IntXLib.FhtMultiplicationException. UnityEngine.Debug:LogError(Object) PostProcessWinRT:RunReferenceRewriter() (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessWinRT.cs:989) PostProcessWinRT:Process() (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessWinRT.cs:211) UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

So, it's in the IntXLib. How can I fix this?

Thanks

[TwoTenPvP]

You are not supposed to include the IntXLib dll if you are not using encryption. Use the DISABLE_CRYPTOGRAPHY directive and the DLL will not be included automatically (on the latest commit as of now, that is). If you are on a older version without the DISABLE_CRYPTOGRAPHY option, just don't move the IntXLib dll over.

[IoannisKaragiannis]

If I don't move the IntXLib.dll over, I get these kind of errors when I build the project in Unity

Reference rewriter: Error: method System.Reflection.FieldInfo System.Type::GetField(System.String,System.Reflection.BindingFlags) doesn't exist in target framework. It is referenced from MLAPI.dll at System.Void MLAPI.NetworkingManagerComponents.Binary.BitWriter::.cctor(). UnityEngine.Debug:LogError(Object) PostProcessWinRT:RunReferenceRewriter() (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessWinRT.cs:989) PostProcessWinRT:Process() (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessWinRT.cs:211) UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

Reference rewriter: Error: method System.Boolean System.Reflection.MemberInfo::IsDefined(System.Type,System.Boolean) doesn't exist in target framework. It is referenced from MLAPI.dll at System.Void MLAPI.MonoBehaviours.Core.NetworkedBehaviour::CacheAttributedMethods(). UnityEngine.Debug:LogError(Object) PostProcessWinRT:RunReferenceRewriter() (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessWinRT.cs:989) PostProcessWinRT:Process() (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessWinRT.cs:211) UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

[TwoTenPvP]

Well. Seems like Reflection has a different API in .NET Core. I will get this sorted shortly.

[TwoTenPvP]

Actually, It seems like Unity doesn't have the reflection stuff at all. Could you provide me with the following:

Your

• Scripting Runtime Version
• Scripting Backend
• Api Compatibility Level

[IoannisKaragiannis]

• Scripting Runtime Version: Experimental (.NET 4.6 Equivalent)
• Scripting Backend: .NET
• Api Compatibility Level: .NET 4.6

[TwoTenPvP]

Is it only GetField and IsDefined that fails?

[IoannisKaragiannis]

Reference rewriter: Error: method System.Boolean System.Reflection.MethodInfo::op_Inequality(System.Reflection.MethodInfo,System.Reflection.MethodInfo) doesn't exist in target framework. It is referenced from MLAPI.dll at System.Void MLAPI.MonoBehaviours.Core.NetworkedBehaviour::OnSyncVarUpdate(System.Object,System.Byte).

Reference rewriter: Error: method System.Reflection.MethodInfo System.Type::GetMethod(System.String,System.Reflection.BindingFlags) doesn't exist in target framework. It is referenced from MLAPI.dll at System.Void MLAPI.MonoBehaviours.Core.NetworkedBehaviour::SyncVarInit().

Reference rewriter: Error: method System.Boolean System.Reflection.MemberInfo::IsDefined(System.Type,System.Boolean) doesn't exist in target framework. It is referenced from MLAPI.dll at System.Void MLAPI.MonoBehaviours.Core.NetworkedBehaviour::SyncVarInit().

Reference rewriter: Error: method System.Boolean System.Reflection.MemberInfo::IsDefined(System.Type,System.Boolean) doesn't exist in target framework. It is referenced from MLAPI.dll at System.Void MLAPI.MonoBehaviours.Core.NetworkedBehaviour::CacheAttributedMethods().

Reference rewriter: Error: method System.Reflection.MethodInfo[] System.Type::GetMethods(System.Reflection.BindingFlags) doesn't exist in target framework. It is referenced from MLAPI.dll at System.Void MLAPI.MonoBehaviours.Core.NetworkedBehaviour::CacheAttributedMethods().

Reference rewriter: Error: method System.Void System.Array::Copy(System.Array,System.Int64,System.Array,System.Int64,System.Int64) doesn't exist in target framework. It is referenced from MLAPI.dll at System.Byte[] MLAPI.NetworkingManagerComponents.Binary.MessageChunker::GetMessageUnordered(System.Collections.Generic.List`1<System.Byte[]>&,System.Int32).

Reference rewriter: Error: method System.Boolean System.Reflection.MemberInfo::IsDefined(System.Type,System.Boolean) doesn't exist in target framework. It is referenced from MLAPI.dll at System.Boolean MLAPI.NetworkingManagerComponents.Binary.BinarySerializer/<>c4`1::b4_1(System.Reflection.FieldInfo).

[TwoTenPvP]

Fixing these problems to comply with .NET Standard 1.6 seems to be fairly tricky. Please build against IL2CPP instead as that has no problems.

(One issue seems to be that the Mono Linker doesn't like the MLAPI's debug sybmbols. Just remove them if that is the case)

[IoannisKaragiannis]

I did as you suggested. Now the project successfully builds in Unity. I then open the project in Visual Studio 2017 and when I try to deploy it on my Hololens device I get the following warnings:

1>c:\users\eakoria\documents\myunityprojects\mlapi-hololens-client\hololensapp\il2cppoutputproject\source\il2cppoutput\genericmethods1.cpp(39000): warning C4717: 'NetworkedBehaviour_SendToClient_TisRuntimeObject_m4039005990_gshared': recursive on all control paths, function will cause runtime stack overflow 1>c:\users\eakoria\documents\myunityprojects\mlapi-hololens-client\hololensapp\il2cppoutputproject\source\il2cppoutput\genericmethods1.cpp(39022): warning C4717: 'NetworkedBehaviour_SendToClients_TisRuntimeObject_m813968917_gshared': recursive on all control paths, function will cause runtime stack overflow 1>c:\users\eakoria\documents\myunityprojects\mlapi-hololens-client\hololensapp\il2cppoutputproject\source\il2cppoutput\genericmethods1.cpp(39044): warning C4717: 'NetworkedBehaviour_SendToClients_TisRuntimeObject_m4187033132_gshared': recursive on all control paths, function will cause runtime stack overflow 1>c:\users\eakoria\documents\myunityprojects\mlapi-hololens-client\hololensapp\il2cppoutputproject\source\il2cppoutput\genericmethods1.cpp(39296): warning C4717: 'NetworkingManager_SendToClient_TisRuntimeObject_m1378064265_gshared': recursive on all control paths, function will cause runtime stack overflow 1>c:\users\eakoria\documents\myunityprojects\mlapi-hololens-client\hololensapp\il2cppoutputproject\source\il2cppoutput\genericmethods1.cpp(39152): warning C4717: 'NetworkedBehaviour_SendToClientTarget_TisRuntimeObject_m4242550098_gshared': recursive on all control paths, function will cause runtime stack overflow 1>c:\users\eakoria\documents\myunityprojects\mlapi-hololens-client\hololensapp\il2cppoutputproject\source\il2cppoutput\genericmethods1.cpp(39338): warning C4717: 'NetworkingManager_SendToClients_TisRuntimeObject_m12938906_gshared': recursive on all control paths, function will cause runtime stack overflow 1>c:\users\eakoria\documents\myunityprojects\mlapi-hololens-client\hololensapp\il2cppoutputproject\source\il2cppoutput\genericmethods1.cpp(39109): warning C4717: 'NetworkedBehaviour_SendToClientsTarget_TisRuntimeObject_m2190847349_gshared': recursive on all control paths, function will cause runtime stack overflow 1>c:\users\eakoria\documents\myunityprojects\mlapi-hololens-client\hololensapp\il2cppoutputproject\source\il2cppoutput\genericmethods1.cpp(39317): warning C4717: 'NetworkingManager_SendToClients_TisRuntimeObject_m649815326_gshared': recursive on all control paths, function will cause runtime stack overflow

However I manage to deploy the game on Hololens

This is what I do now:

1) Start Server on my desktop 2) Start client-A on my laptop 3) Start client-B on Hololens

For some reason when client-B from Hololens joins the game, the Server destroys client-A and he only displays client-B. Client-A on his local screen cannot see client-B and he moves locally without managing to send his position updates to anybody.

Any idea what could be wrong? Are you sure the IL2CPP is safe?

[TwoTenPvP]

I havent tested il2cpp. I just know it compiles. I’ll try to get a build ready that is fully unity compliant. Seems like living on the edge with .NET features is not a good idea.

[TwoTenPvP]

I am unsure about why the IL2CPP build detects an infinite recursion. I guess this has something to do with the send overloads since they call each other to provide the same functionality in less code. To be honest, there isn't too much I can do. The project depends on reflection in many ways. The SyncedVar and the attribute messaging system is built on it.

I will have to rethink what to do here.

[IoannisKaragiannis]

I managed to deploy the application on HoloLens. However, I now get the behavior I had described previously. Once the HoloLens (Client-B) joins the game I can instantaneously see Client-A everywhere, but then Client-A is destroyed on the Server's side and he can only move locally on his own world. I can no longer connect to the Server with Client-A. This is very strange and I have no clue where to start debugging it. I cannot really explain it better. When I tried with the HoloLens-emulator it was working fine but with the Hololens device it seems to be failing.

[TwoTenPvP]

I am having a hard time understanding the problem. Please describe EVERY step. And during EVERY step, describe EXACTLY what EVERYONE can see. Example:

Player 1 starts Host. He can see his client and interact fine. Player 2 starts client and connects. Host can see 2 and 2 can see host. Everything works fine.

[IoannisKaragiannis]

Ok let me give it a better try.

I have one Server on my desktop without a Host. I call this project Server. I have 2 players on 2 different laptops. The players' applications are identical and I call the Client. I have a third player on my Hololens device. For your information, when I built the client-app for the HoloLens I selected the IP2CPP as my backend scripting. I have also selected the InternetClientServer capability under the Publishing Settings. I'm using MLAPI v1.3.0.

These are the steps:

1. Server starts in my Desktop (no Host). I spawn a cube in the middle of my Scene.
2. Player-1 starts Client and connects (laptop-1). He sees the cube and he moves around. The Server can see Player-1. Everything is fine up to this point.
3. Player-2 starts Client and connects (laptop-2). He sees the cube and Player-1 and he moves around. The Server and Player-1 can see Player-2. Everything is fine up to this point.
4. Player-3 starts Client and connects (HoloLens headset). He sees the cube and only instantaneously Player-1 and Player-2. Player-1 and Player-2 are no longer part of the HoloLens scene. Player-1 and Player-2 are destroyed on the Server's side. Server can only see Player-3. Player-1 can only see himself moving around without being able to touch the cube. Player-2 can only see himself without being able to interact with the cube. So, it's like Player-3 is so dominant that he forces the Server to destroy all the rest.

Could it be that the networking manager is not working as expected when we use the IP2CPP? I'm just guessing. Maybe I should try to build both the Server and the Client as UWP with the IP2CPP option and then deploy them on the respective local machines via Visual Studio. Then I should deploy the Client on the Hololens device and see what happens. It could be that if one application is built with .NET and the other with IP2CPP they cannot really work together. Again, just guessing.

[IoannisKaragiannis]

Hey,

I'm back with some answers here.

So far I have been having the Server running on my windows-10 desktop and the 2 remote clients on windows 7 laptops. Then, I was connecting the 3rd client (HoloLens) via and Adhoc network to my windows-10 desktop which was running the Server. For some reason when the 3rd client was joining the game, something was messed up with the sockets I guess, and the Server could not listen to the other remote clients any more. It's probably some configuration in the Adhoc network that creates this problem. I have no explanation for this yet. Any insight is more than welcome.

Now I decided to move the Server on one of my windows-7 laptops. Then I start the 1st clients on the other windows-7 laptop, the 2nd client on the windows-10 desktop, and the 3rd one on HoloLens. Now everything works fine. Everyone sees everybody and they are all happy. This was very tricky. I still can't get it why the Adhoc network is causing that problem.

[IoannisKaragiannis]

Have you managed to successfully build your project with IP2CPP for UWP but keeping cryptography in MLAPI? Haven't really tested if I still get these errors I was getting (the System.Security.Cryptography errors) when trying with .NET as the scripting backend.

[TwoTenPvP]

I have no time currently to investigate in that as i'm finishing my gymnasium education. But I believe cryptography should work with IL2CPP as I think it still uses the Mono runtime and not .NET Core.

[IoannisKaragiannis]

No worries. Good luck with your exams pal. And don't get too drunk on your graduation. Cheers

[TwoTenPvP]

MLAPI >= 6.0 "Lite" version should work with HoloLens. And I am fairly certain that it will work with the standard version. But this is no longer an issue anyhow.