Unity 2019 UWP Hololens compatibility

[sBailey5449]

The README instructions mention using the 2017 version of MRTK, which is not compatible with Unity 2019.x out of the box. MRTKv2 works fine but does not include the Newtonsoft.json.dlls needed by this repository (at least for commits before the 2019 update on master). I noticed that by default the Newtonsoft.json.dll included in this repo is now set to "any platforms" which leads me to think the dependency on MRTK 2017 has been removed? When I try to build for the Hololens using Unity 2019.1.7f1, the latest version of MRTKv2 (RC2.1), and the latest commit on master for this repository (4ccf45fc) I see the following error when running in debug mode on the Hololens. Everything works fine in the Editor:

NotSupportedException: System.Reflection.Emit.DynamicMethod::.ctor
  at System.Reflection.Emit.DynamicMethod..ctor (System.String name, System.Type returnType, System.Type[] parameterTypes, System.Type owner, System.Boolean skipVisibility) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Utilities.DynamicReflectionDelegateFactory.CreateDynamicMethod (System.String name, System.Type returnType, System.Type[] parameterTypes, System.Type owner) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Utilities.DynamicReflectionDelegateFactory.CreateGet[T] (System.Reflection.FieldInfo fieldInfo) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Utilities.ReflectionDelegateFactory.CreateGet[T] (System.Reflection.MemberInfo memberInfo) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.DynamicValueProvider.GetValue (System.Object target) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CalculatePropertyValues (Newtonsoft.Json.JsonWriter writer, System.Object value, Newtonsoft.Json.Serialization.JsonContainerContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonProperty property, Newtonsoft.Json.Serialization.JsonContract& memberContract, System.Object& memberValue) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject (Newtonsoft.Json.JsonWriter writer, System.Object value, Newtonsoft.Json.Serialization.JsonObjectContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract collectionContract, Newtonsoft.Json.Serialization.JsonProperty containerProperty) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue (Newtonsoft.Json.JsonWriter writer, System.Object value, Newtonsoft.Json.Serialization.JsonContract valueContract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerProperty) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize (Newtonsoft.Json.JsonWriter jsonWriter, System.Object value, System.Type objectType) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonSerializer.SerializeInternal (Newtonsoft.Json.JsonWriter jsonWriter, System.Object value, System.Type objectType) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonSerializer.Serialize (Newtonsoft.Json.JsonWriter jsonWriter, System.Object value, System.Type objectType) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonConvert.SerializeObjectInternal (System.Object value, System.Type type, Newtonsoft.Json.JsonSerializer jsonSerializer) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonConvert.SerializeObject (System.Object value, System.Type type, Newtonsoft.Json.JsonSerializerSettings settings) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonConvert.SerializeObject (System.Object value) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.RosSocket.Serialize[T] (T obj) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.RosSocket.Send[T] (T communication) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.RosSocket.Subscribe[T] (System.String topic, RosSharp.RosBridgeClient.SubscriptionHandler`1[T] subscriptionHandler, System.Int32 throttle_rate, System.Int32 queue_length, System.Int32 fragment_size, System.String compression) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.Subscriber`1[T].Start () [0x00000] in <00000000000000000000000000000000>:0 
...
Rethrow as JsonSerializationException: Error getting value from 'topic' on 'RosSharp.RosBridgeClient.Subscription'.
  at Newtonsoft.Json.Serialization.DynamicValueProvider.GetValue (System.Object target) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CalculatePropertyValues (Newtonsoft.Json.JsonWriter writer, System.Object value, Newtonsoft.Json.Serialization.JsonContainerContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonProperty property, Newtonsoft.Json.Serialization.JsonContract& memberContract, System.Object& memberValue) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject (Newtonsoft.Json.JsonWriter writer, System.Object value, Newtonsoft.Json.Serialization.JsonObjectContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract collectionContract, Newtonsoft.Json.Serialization.JsonProperty containerProperty) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue (Newtonsoft.Json.JsonWriter writer, System.Object value, Newtonsoft.Json.Serialization.JsonContract valueContract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerProperty) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize (Newtonsoft.Json.JsonWriter jsonWriter, System.Object value, System.Type objectType) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonSerializer.SerializeInternal (Newtonsoft.Json.JsonWriter jsonWriter, System.Object value, System.Type objectType) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonSerializer.Serialize (Newtonsoft.Json.JsonWriter jsonWriter, System.Object value, System.Type objectType) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonConvert.SerializeObjectInternal (System.Object value, System.Type type, Newtonsoft.Json.JsonSerializer jsonSerializer) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonConvert.SerializeObject (System.Object value, System.Type type, Newtonsoft.Json.JsonSerializerSettings settings) [0x00000] in <00000000000000000000000000000000>:0 
  at Newtonsoft.Json.JsonConvert.SerializeObject (System.Object value) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.RosSocket.Serialize[T] (T obj) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.RosSocket.Send[T] (T communication) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.RosSocket.Subscribe[T] (System.String topic, RosSharp.RosBridgeClient.SubscriptionHandler`1[T] subscriptionHandler, System.Int32 throttle_rate, System.Int32 queue_length, System.Int32 fragment_size, System.String compression) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.Subscriber`1[T].Start () [0x00000] in <00000000000000000000000000000000>:0 
...

After doing some poking around it seems like this might be an issue related to the IL2CPP scripting backend ( Unity 2019.1 deprecates the .NET scripting backend). Not sure if there's a few steps missing from the README, or if the Newtonsoft.json.dll needs to be updated. Any help is greatly appreciated!

[sBailey5449]

Tried using MRTKv2 with current tip of master, with MRTK 2017's bundled versions of Newtonsoft.json.dlls but still had IL2CPP related problems. Current workaround is to use latest commit on UWP branch with unity 2018.3 and MRTK 2017, but not sure if this the only/best option.

[MischaRo]

Hi @sBailey5449 I also tried to build the project for Hololens MRTKv2 using Unity 2019.1.0f2 and got the same issues. Have you made any progress with the MRTKv2 and Newtonsoft.json.dlls / IL2CPP related problems?

[thommi1609]

Same here, can't get it to work. Any news on this issue?

[curthenrichs]

I have had success with using the AOT version of Newtonsoft as recommended in issue #2269 in the mixed-reality-toolkit: https://github.com/Microsoft/MixedRealityToolkit-Unity/issues/2269

https://assetstore.unity.com/packages/tools/input-management/json-net-for-unity-11347

Notes:

• I needed to disable the Netwonsoft.json in RosSharp/Plugins for all platforms.
• When building the application it threw 21 errors but finished the build.

I have only tested this approach on a Hololens v1 with a simple application subscribing to joint state topic to update a URDF robot.

[tgroechel]

Can also confirm @curthenrichs fix works for Hololens 2 + BSON, have tried with laser scan and image messages for subscribing and publishing twist

[sBailey5449]

I haven't had any success using @curthenrichs approach. It will build despite throwing a few dozen errors, but when running on HoloLens1 I just see Exception thrown at<memory location> in <application name>: Microsoft C++ exception: Il2CppExceptionWrapper at memory location <memory location>. over and over. Trying to deserialize a PointCloud2 message, using Unity 2019.3.6f1. Everything works perfectly in the editor.

[tgroechel]

@sBailey5449 Try building for debug and running which should give you better info in visual studio.

My guess is you are getting NaN or Inf values. I had this issue with laser messages and isolated by using other more stable messages to confirm they work in the Hololens (e.g. Odom). My fix was to use BSON serialization instead of JSON as well as check for NaN and Inf inside of C# scripts.

[sBailey5449]

@tgroechel I should have given more information in my last post, that error is from running in debug mode and occurs each time I try to deserialize a PointCloud2 message. Stepping through the generated c++ code I can see that the error occurs when trying to deserialize a message, so it's coming from the Newtonsoft.Json code and not ROS#. Using Unity 2018.4 I can make a .NET backend build that works fine on the HoloLens1, but as soon as I switch to IL2CPP I see the problem I described above (even when using the AOT Newtonsoft.json dll mentioned above). I suspect that other types of messages will probably work but I haven't tried yet. I also tried using BSON, but I still have the same errors.

The real problem seems to be that we need a modern, UWP-compatible version of the newtonsoft.json dependency. The JSON.Net for Unity asset was last updated in 2017 and uses an older version of the newtonsoft.json code (looks like version 8.0.0.0), ROS# currently depends on a more recent version (looks like version 12.0.0.0).

[aioniosfoititis]

I have been using the AOT Json files with 2019.2.11f unity and works on hololens 1.I have also noticed that for 2019.3 unity this setup is not working.

Maybe stick with 2019.2?

[dnlwbr]

For me @curthenrichs approach (AOT Json files) worked with Unity 2018.4.22f and IL2CPP as well. At least in principle. I successfully subscibed to CompressedImage messages and published Twist messages. However when I try to subscribe to Odometry I get the following error:

NullReferenceException: Object reference not set to an instance of an object.
  at RosSharp.RosBridgeClient.Communicator.GetRosName[T] () [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.Subscriber`1[T]..ctor (System.String id, System.String topic, RosSharp.RosBridgeClient.SubscriptionHandler`1[T] subscriptionHandler, RosSharp.RosBridgeClient.Subscription& subscription, System.Int32 throttle_rate, System.Int32 queue_length, System.Int32 fragment_size, System.String compression) [0x00000] in <00000000000000000000000000000000>:0 
  at RosSharp.RosBridgeClient.RosSocket.Subscribe[T] (System.String topic, RosSharp.RosBridgeClient.SubscriptionHandler`1[T] subscriptionHandler, System.Int32 throttle_rate, System.Int32 queue_length, System.Int32 fragment_size, System.String compression) [0x00000] in <00000000000000000000000000000000>:0 
  at Calibration.Start () [0x00000] in <00000000000000000000000000000000>:0

What could be the reason for that? In the Unity Simulation everything works fine.

[EricVoll]

This person created a Unity compatible version of Newtonsoft.Json I just tested it with Unity 2019.3.7f1 and it did build and work without a problem for Windows Standalone and UWP. Thus it should also be HoloLens compatible. I wonder why I did not find it earlier...

https://github.com/jilleJr/Newtonsoft.Json-for-Unity

As @curthenrichs said, you have to disable the "original" Newtonsoft.json.dll for all platforms.

[dnlwbr]

Yeah, you're right. Thank you very much. It compiles without errors and works for UWP.

But still I have the same problem with the odometry topic as mentioned before. Could this be a bug in Ros#, since it works in Unity and subscribing to CompressedImage works on the device?

[EricVoll]

Yeah about the Communicator class. They use Reflection in the Communicator.GetRosName method. They had to do that, because the Message class does not have the RosMessageName field on it, but every class inheriting from Message implements it on its own.

Generally, Reflection does not work on HoloLens / UWP Builds, because of the AOT compilation. I did read, that Reflection itself shouldn't be a problem and that only the Emit namespace does not work, but I experienced, that everything that has to do with reflection fails.

I fixed this issue in my fork here.: https://github.com/luchspeter/ros-sharp/commit/a7d14016996337bfb769c3118b920e1d8e91220b I received the exact same Exception as you are. In short: Add a RosMessageName : string field to the Message class and then you can easily remove the reflection code. I'd even suggest, that the main repository would also do that, because Reflection is very inefficient, compared to something like this. If the maintainers want, I can create a PR with that change.

[dnlwbr]

Since I am not an expert, I am not sure wether I am able to do that myself without further issues. I would really welcome a PR. Therefore I would be happy if the maintainers agree. Is that possible @dwhit ?

[dnlwbr]

@luchspeter Is there anything important missing in your fork compared to this one? Otherwise, if you just added some features and fixed the issue mentioned above, I will simply use your fork.

[EricVoll]

Well my fork is more or less a re-write of this fork in some parts. The library itself is perfectly usable and simply has more features, but the Unity Part is generally not the same anymore. I rewrote that to be a runtime compatible version of it. So in conclusion: You should be able to (and of course are welcomed to) use the library of my fork, but the Unity parts are very different.

[dnlwbr]

I have imported your fork into my project and unfortunatly I get two errors:

Assets\RosSharp\Scripts\Urdf\Builder\UrdfComponents\UrdfLinkExtensions.cs(35,21): error CS0103: The name 'AttachedDataSynchronizer' does not exist in the current context
Assets\RosSharp\Scripts\Urdf\Builder\UrdfComponents\UrdfLinkExtensions.cs(51,96): error CS0246: The type or namespace name 'AttachedValue' could not be found (are you missing a using directive or an assembly reference?)

Since the maintainers do not respond, is there any chance that instead of a PR, you could apply the said changes to an extra branch in your own fork? I would really appreciate it.

[EricVoll]

Yeah I expected these errors to appear, since the fork is a complete rewrite of some parts to allow ROS# to generate Robots during Runtime. The Unity part of the repository only works with the "new" library as well.

Let me see. Maybe I can simply cherry-pick the right commit. I'll have a look. Are you under time pressure? Life is stressful at my end right now.

[dnlwbr]

No, not very much, it's more important to me that it works without any problems. I'm glad you're taking the time at all. So no stress and thank you very much!

[EricVoll]

@dnlwbr So I created a branch on my fork of this repository and cherry picked the commit that fixed your problem in my case. The branch origins from the current state of this repository, so none of my other changes are in there.

Here's the link https://github.com/luchspeter/ros-sharp/tree/ReflectionFix I don't have Unity installed on my current machine, but if I remember correctly I had to make a small change in the Unity scripts as well. For some classes that accept a Message of "type T" I added the requirement of "T" that it implements an empty constructor. The same thing had to be done in one or two of the Unity scripts.

But you can raise an issue on the repository over there if you can't manage to fix it.

Hope this helps!

[dnlwbr]

@luchspeter Thank you very much! I really appreciate it. I was able to make the small changes in the Unity scripts myself and now I am able to subscripe to odometry as well. Unfortunately the whole thing starts to crash over and over again showing: Ausnahme ausgelöst bei 0x00007FFFBFBA17B8 (vcruntime140d_app.dll) in TestProgramm.exe: 0xC0000005: Zugriffsverletzung beim Schreiben an Position 0x000001D69DCCFFA0. Did you have the same issues? What could be the cause?

[EricVoll]

öh nope I never had that problem. Could you post the full stacktrace? I think the error might appear somewhere else in your application. That should probably be a new issue though.

[dnlwbr]

The only log in debug mode I get is:

Der Thread 0x1700 hat mit Code 0 (0x0) geendet.
Der Thread 0x129c hat mit Code 0 (0x0) geendet.
Ausnahme ausgelöst bei 0x00007FFFD50F39EC in TestProgramm.exe: Microsoft C++-Ausnahme: Il2CppExceptionWrapper bei Speicherort 0x0000009ABB8FD3E0.
Ausnahme ausgelöst bei 0x00007FFFC0FD17B8 (vcruntime140d_app.dll) in TestProgramm.exe: 0xC0000005: Zugriffsverletzung beim Schreiben an Position 0x0000028F1C08FFA0.

But the Exception seems to be triggered in ...\Build\Il2CppOutputProject\IL2CPP\libil2cpp\icalls\mscorlib\System\Buffer.cpp (GameAssembly.dll!il2cpp::icalls::mscorlib::System::Buffer::BlockCopyInternal(Il2CppArray src, int src_offset, Il2CppArray dest, int dest_offset, int count) Zeile 33) in the following block (see mark):

bool Buffer::BlockCopyInternal(Il2CppArray * src, int src_offset, Il2CppArray * dest, int dest_offset, int count)
    {
        IL2CPP_CHECK_ARG_NULL(src);
        IL2CPP_CHECK_ARG_NULL(dest);

        /* watch out for integer overflow */
        if (((uint32_t)src_offset > il2cpp::vm::Array::GetByteLength(src) - count) || ((uint32_t)dest_offset > il2cpp::vm::Array::GetByteLength(dest) - count))
            return false;

        char *src_buf = ((char*)il2cpp_array_addr_with_size(src, Class::GetInstanceSize(src->klass->element_class), 0)) + src_offset;
        char *dest_buf = ((char*)il2cpp_array_addr_with_size(dest, Class::GetInstanceSize(dest->klass->element_class), 0)) + dest_offset;

        if (src != dest)
            memcpy(dest_buf, src_buf, count); // <-- Exception is triggered here
        else
            memmove(dest_buf, src_buf, count); /* Source and dest are the same array */

        return true;

    }

[dnlwbr]

@luchspeter Just to be sure: After building RosSharp I only have to copy RosBridigeClient.dll and RosBridigeClientUWP.dll in my project? No Newtonsoft.Json.dll or any of the other files from the plugins folder right? Should I build with debug, release or master?

[EricVoll]

You should configure the building of these dlls exactly as it is written in the ReadMe of this repository. Then add the normal NewtonSoft.Json of this repository into the same "plugins" folder, but disable all build targets for the newtonsoft library. Then include the "other" newtonsoft library as written above: https://github.com/dwhit/ros-sharp/issues/3#issuecomment-631617300

Adding the "normal" Newtonsoft Library and disabling all build targets solved the compiler warnings in my case during Development stages.

But I suggest you start with an empty project and add components / libraries and build after every single one and then check if something breaks.

Your exception looks like another problem. Typically these dependency/building issues look differently... Maybe you are trying to write to an AssetPath or whatever which is not possible in a built game.

[dnlwbr]

@luchspeter I did everything exactly as you said, and it seems to work now. At least no errors occured so far. I don't know what exactly the problem was. So thank you again for your help and for taking the time!