Improve Unity support

[AArnott]

In response to a comment made by @odysoftware in https://github.com/neuecc/MessagePack-CSharp/issues/352#issuecomment-446877252

One of my favourite wishes would be the compatibility of the library with unity .NET Standard 2.0 ! At the moment due to the use of emit - this is sadly not possible :)

I know ~nothing when it comes to Unity. From the Unity support I found in this library while retrofitting its build/test system, it seems that the distribution for Unity is source code rather than a library. @odysoftware Is that a non-requirement at this point, if Unity supports .NET Standard 2.0? I'd be very happy to rip out the dedicated Unity source distribution stuff in the repo and just retain its .netstandard2.0 build for Unity support.

Regarding ref-emit use, I see that it must not be required since alternative code exists to support Unity. If I can figure out a way to make it a runtime check for Unity and skip it, so that we can distribute a single netstandard2.0 binary that works on and off Unity, that would be preferable to its current state, I think. Unity can consume nuget packages, I presume?

[odysoftware]

The problem is that Unity is multi platform so everything you create needs to be AOT compatible. So that your code will work on platforms like mac/iOS or whenever unity is converting your IL code using the IL2CPP to c++.

So basically as long as we don't use any reflection etc. we are safe to use .NET Standard 2.0 or as most people now .NET 4.5 within unity.

As far as I know messagepack is not using any reflection - because it works totally fine on all platforms with unity. So I am also not too much into the technical details of unity and the il2cpp - since I am a sole C# developer - maybe someone can give a more clear answer here where the problems could rely...

[odysoftware]

Oh and no unity does not use nuget. Its a complete independent environment (a game engine). But it can consume your .NET 4.5 or .net standard 2.0 libraries for use.

[mikkoairaksinen]

We are currently using MessagePack with Unity in a mobile game context. The way the library currently circumvents using the System.Reflection.Emit namespace is by providing a code generator that you must run manually to generate the resolvers your code needs (the mpc.exe part in the front page). The code generator has some limitations, such as supporting generic types so in addition to this you must implement formatters for generic types yourself (having done this though, I think there should be a way to generate even the generic formatters).

AFAIK there have been some proposals in the past of the original repository to turn the manual mpc.exe execution into an MSBuild build step instead (like the Microsoft Orleans framework does for example), but that runs into the issue that Unity regenerates the solution and csproj files on it's own constantly so any changes to those won't be permanent. It would be awesome if there was some way around this and the code generation could be automatic part of the Unity build.

For the nuget part, unfortunately Unity's package management is not great to say the least, so the current way to use .NET Standard 2.0 libraries is to copy paste the .cs files or DLL into the project manually and then update manually when necessary.

[mikkoairaksinen]

Also, Unity still has limited support for latest C# features (C#6 compatibility was reached early this year and only three days ago was C# 7 made available with Unity 2018.3) so any of the suggested changes where the library would rely on Span for example would break older Unity versions compatibility instantly.

[AArnott]

Thanks. This is great info to have.

@mikkoairaksinen regarding the C# feature support, I suppose that's only a problem if we use source sharing. If we ship a unity-compatible DLL, that wouldn't be an issue I guess, so long as that DLL only references other assemblies that Unity supports. Is that right?

@odysoftware MessagePack certainly does use reflection, at least in the typical case. It's based on attributed types and members, which can only be retrieved via reflection. AFAIK all platforms allow reflection -- it's the code generating Reflection.Emit code that is rejected by AOT platforms. Does the MCP.exe tool completely eliminate even reflection that looks for attributes?

[odysoftware]

Yeah sure I guess I was explaining it a bit confusing. Surely it uses reflection but thats why we have the mpc.exe which generates the code for AOT platforms as you have already explained :)

Either way I wonder how difficult it would be to transition the messagepack dll to be compatible to .NET standard 2.0 - so we could switch the unity project from .net 4.5 to .net standard 2.0 - which would be great since only messagepack is giving errors.

[mikkoairaksinen]

@AArnott Here's a compiled doc for (some of) the restrictions of Unity https://docs.unity3d.com/Manual/ScriptingRestrictions.html

I'm not super well versed in C# versions and runtime support, but another thing to take into account is that Unity uses it's own custom Mono-based runtime that is (roughly) equivalent of .NET 4.6 at this point, but still custom, so some things might not work as assumed. It seems to be possible to use at least some C#7 features with external DLLs like you mentioned, but there's a chance that everything is not compatible.

[BrennanConroy]

I filed an issue a while back for not using Reflection.Emit and neuecc started experimenting with a fix https://github.com/neuecc/MessagePack-CSharp/commit/bff2ea3eb2051614f1d9cd336f1a4fb1deafe535

[jsteinich]

We actually use the main MessagePack dll in Unity. Our MessagePack objects exist in a library project that is copied as a dll to Unity. We are running mpc to generate resolvers/formatters. This mostly works, but there are a few issues.

• Unity detects the references to System.Reflection.Emit and spits out an error, but since that code path doesn't hit at runtime it works correctly. We specify a resolver to ensure that.
• We have some developers on mac and they are currently unable to run mpc.
• NuGet support is pretty poor, but manageable when not updating libraries frequently.

Unity is pretty particular about assembly versions matching, so from our perspective having a single dll that varies at runtime is preferable.

[AArnott]

Great info. Thanks for sharing, folks. Keep it coming. :)

[AArnott]

Unity detects the references to System.Reflection.Emit and spits out an error, but since that code path doesn't hit at runtime it works correctly. We specify a resolver to ensure that.

So a dedicated DLL that omits the Ref-Emit calls would be preferable as it would fix the errors.

Unity is pretty particular about assembly versions matching, so from our perspective having a single dll that varies at runtime is preferable.

That contradicts what I gathered from your prior comment. What if there were two MessagePack.dll's, with the same assembly name and version, such that for a runtime they would be interchangeable, but one is built for Unity and is missing the Ref-Emit calls? That's basically what multi-targeting is for. I don't know if unity has a TFM in NuGet, but we might make one up.

[jsteinich]

Let me try rephrasing...

• There are some libraries where Unity support is provided as a completely different assembly to the normal version. That approach doesn't really work well when an intermediate library is used since Unity will just see that a referenced assembly is missing even though compatible classes may exist.
• We have refrained from using the MessagePack Unity code for pretty much that same reason, although we probably could have tried building a dll that matched the non-Unity version.
• Unity expects assemblies to match version exactly where as other systems can have non-exact matches. For example if library A references version X of library C and library B uses version X+1, then even if X+1 is compatible with X, Unity won't be happy with it.

Using multi-targeting to have a Unity version with the same assembly name and version sounds great. There may be some that would still prefer to add a Unity package, but I'd rather have a dll. Unity will sometimes compile things differently from source vs a dll, but with the latest version adding Roslyn support that may no longer be true.

[neuecc]

Sorry for late response(again...!) I've updated master code because for my new project ( https://github.com/Cysharp/MagicOnion ) , it requires MessagePack for C# and Unity. Add .NET Standard 2.0 support and code generator (mac, linux support, but still incomplete).

Well, Unity needs four correspondences.

• .NET 3.5 Equivant (for legacy Unity)
• .NET 4.6 Equivant (for new Unity)
• .NET Standard 2.0 (for new Unity)
• IL2CPP (Unity AOT Compilation)

Basically, libraries for Unity are provided at the source code level. It is because it needs to be judged at compile time in order to deal with various kinds of environments as above. Also, think that libraries provided with NuGet can hardly be used. Therefore it is not very appropriate to bring Span etc to reference to Unity by reference.

.NET 3.5, .NET 4.6 is easy, the current one and the same thing as for the .NET Framework will work. In .NET Standard 2.0, dynamic code generation related can not be used in .NET 4.6. At the time of execution, dynamic code generation throws an exception at IL2CPP.

Because I prioritize the best performance in the IL2CPP environment for performance reasons, For IL2CPP, a pre-code generation approach is taken (mpc.exe, analyzed by Roslyn to prepare the equivalent of dynamic code generation in advance). Reflection based on IL2CPP will work, As it is imagined that performance is slow and probably poor results with generic base generation such as T[], So implementation is not completed.

The server's .NET and the client's Unity refer to a "different" MessagePack Bringing serious problems in code share, Currently, I am processing by SharedProject, but it would be best if I could do with a single DLL.