Mixed Reality Collaboration question

[astaikos316]

Have a couple of questions. First, trying to build a remote collaboration capability and wondering how to access the data channel in Unity in order to send not only the Hololens spatial mesh but also allow sending holograms/annotations to devices. Second, is there a way to use a Websockets signalling server instead of the http server?

[eirikhollis]

I don't know about the data channels, but websocket signalling is both possible and works quite nice. You would have to create your own signaller class, and probably modify PeerConnection and Local/Remote Video/Audio Source scripts. Depends on how much you want to re-use.

[djee-ms]

Hi @astaikos316,

Yes as @eirikhollis explained you can use any signaling solution by deriving from the Unity Signaler abstract base component and implementing the various bits of the interface, and most notably the ISignaler interface. You can look at the NodeDssSignaler for an example.

About the data channels, these are just raw streams of bytes so I couldn't see a good use for a Unity component. Instead you can directly access the underlying C# PeerConnection object from the Unity component's Peer property and use methods like AddDataChannelAsync() to create a new data channel.

[djee-ms]

Note: you must create data channels after the PeerConnection is initialized, so the underlying native object is created, but you must also create at least one data channel before the connection is established, because otherwise the SCTP handshake is not performed and data channels cannot be added later. This is a limitation of the underlying implementation, possibly even of the standard itself (not sure). Other data channels can be added later after the connection is established.

[pablomarcos]

Hi @djee-ms , any chance to create a short example about how to do that for sending a simple string of text between the clients? It would be very useful to understand how to do it correctly. Thanks a lot.

[djee-ms]

@pablomarcos > You can look at how it's done in the TestAppUWP sample, which does exactly that (uses a data channel for sending text between the peers to simulate a text chat):

https://github.com/microsoft/MixedReality-WebRTC/blob/5d70abde752f8ec38645eac6a0106ac97d8e27b7/examples/TestAppUwp/MainPage.xaml.cs#L398-L416

[djee-ms]

@astaikos316 did you get the answers to all your questions? Can I close this issue?

[astaikos316]

Been having some trouble wrapping it into the Unity example as we are trying to develop collaborative mixed reality application. Trying to trace through the xaml file you pointed out running into trouble on how to expand it into the unity example

[djee-ms]

Maybe I am misunderstanding what you are trying to do and it is more complex than what I imagine. But I would assume that if you need a data channel to send custom data like spatial mesh, you can:

1. Take your WebRTC.Unity.PeerConnection Unity component, and access its Peer property, which gives you a WebRTC.PeerConnection C# object from the underlying .NET library.

2. Call when needed PeerConnection.AddDataChannelAsync(). If you know in advance that you need a data channel (and you seem to do), you might as well do that call just after the OnInitialized event fired, when the PeerConnection is ready for use but the connection has not yet started (CreateOffer() not called). This ensures that DTLS will perform its handshake and data channel work.

3. Send and receive data through that channel with the SendMessage() method and MessageReceived event.

There's not much more to data channels. It's just a raw stream of bytes, with 2 actions (send and receive). Or did I misunderstood your use case?

[pablomarcos]

In my case I have the following problem: sending a string of text from HoloLens to Unity editor is working correctly. Sending a string from Unity Editor to HoloLens makes the Holo app crash without a clear error message. It just crash after receiving the message and deserialize the byte array. I need to dig more into the problem...

[pablomarcos]

This is the error I'm getting: _CRT_ERROR caught: '''(0) : Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention.

Please follow the link for the full Log from HoloLens: https://pastebin.com/z49zsbV1

[djee-ms]

@pablomarcos that's a P/Invoke function call failing, where the C# and C++ sides have different conventions. MixedReality-WebRTC uses __stdcall because Unity forces that convention when transpiling with IL2CPP.

I would bet on a stale library on the HoloLens. The fact that the callstack seem to have recursive calls to mrsMemCpyStride is very suspicious, because this function is not recursive, so it looks like a corrupted callstack. Although that could also be a bug to be honest, since the stack corruption is likely the result and not the cause.

I would try to log the string immediately after receiving it, to make sure it's not garbage. Beyond that, identifying which function exactly is failing will greatly help debugging.

[pablomarcos]

@djee-ms thanks for your help. I can deserialize the message and it's correct. It contains the correct string I have sent from Desktop to the HoloLens but after do it is when the HoloLens crash. That's the reason it is so strange...because the message is received correctly, but for some reason after some seconds it crash. I'm not sure how to continue debugging the problem, so any help from your side would be highly appreciated.

[pablomarcos]

@djee-ms After debugging the app with VS debugger I got this method as the cause of the crash (image attached)

[djee-ms]

Ok my bad I just saw MemCpy is called from DataChannel.OnMessageReceived() and is passing a C# ulong (64-bits), whereas the C/C++ signature uses size_t which is 32-bit on HoloLens. Logged #63, will fix now.

[djee-ms]

@pablomarcos > Fixed in fb4a230ab39cf943f581b8caa4592d0f8029401a, can you try again with that commit please? I am pretty sure that was it.

[astaikos316]

@pablomarcos - can you provide some insight on how you managed to set things up in unity to get the data channel and send text? I have built a chat window in unity and trying to send things over the data channel, but I can’t seem to figure out how to set things up like djee-ms described earlier

[pablomarcos]

@pablomarcos > Fixed in fb4a230, can you try again with that commit please? I am pretty sure that was it.

Gonna try and I will tell you. Thanks a lot!

[pablomarcos]

@pablomarcos - can you provide some insight on how you managed to set things up in unity to get the data channel and send text? I have built a chat window in unity and trying to send things over the data channel, but I can’t seem to figure out how to set things up like djee-ms described earlier

Sure let me get into the office and I will write you some lines about how we did it.

[pablomarcos]

@djee-ms I'm getting the following errors when building the libraries from the VS 2019 project on the root folder. (It appears to be the last changes from 'size_t' to 'uint64_t'). Despite the errors the libraries are generated and copied into Unity's folder.

[djee-ms]

Ah this is an error in the unit test, will fix, thanks!

[djee-ms]

Fixed with 5b4956f7c163d2cdd55e637caa097ee210cc8e79.

[astaikos316]

@djee-ms I have been trying to follow your process flow in the unity sample, but I am having trouble based on the documentation. I have set up a chat box in the sample to test the data channel messaging but cannot get it to work at all. @pablomarcos if you have a chance to provide some insight on how you implemented your test case it would be greatly appreciated.

[djee-ms]

What is the problem @astaikos316? Can you describe what you are doing, or do you have some sample code we can look at? What do you mean by "not working at all", what piece is not working?

[astaikos316]

@djee-ms I am starting with the Video Chat Demo Unity Scene. I have set up a Chat Window to try and send and receive data. I built an Empty Chat Manager game object and attached a script to it where I try to add the data channel on the peer connection oninitialized() event, but I cannot figure out how to verify the data channel state. Then I try to call sendmessage() and get object reference not set to an instance of an object error. I am just trying to understand a simple unity setup to accomplish this type of task.

[pablomarcos]

@astaikos316 Sorry I was out of the office for several days. So following @djee-ms advice this is what I did... First thing is to add the new data channel we will use for sending our text messages (or wathever thing you want). I'm doing this on the PeerConnection.cs script, exactly on method OnPostInitialize() after the line OnInitialized.Invoke();

And this is the code I'm using for creating the data channel:

_nativePeer.AddDataChannelAsync(ChatChannelID, "chat", true, true).ContinueWith((prevTask) =>
{ 
    if (prevTask.Exception != null)
    {
        throw prevTask.Exception;
    }
    var newDataChannel = prevTask.Result;

    _mainThreadWorkQueue.Enqueue(() =>
    {
        _chatDataChannel = newDataChannel;
        _chatDataChannel.MessageReceived += ChatMessageReceived;
        Debug.Log("DataChannel for CHAT created successfully.");
    });
});

Note that ChatMessageReceived must run on the main thread so the first idea was to use the queue that already exists. The problem here is that after all it seems not to be running on the main thread so when I try to update my UI on my ChatMessageReceived function the app is crashing. So for solving this I have to be sure that I'm modifying the UI from the main thread. I did this storing the message received on a temporal variable and checking on the Update() method if this variable has a new value, then I write it on my UI.

private void ChatMessageReceived(byte[] message) { string text = System.Text.Encoding.UTF8.GetString(message); textToProcess = text; }

And as you can imagine on my Update function: if (textToProcess != String.Empty) { textReceivedMesh.text = textToProcess; textToProcess = String.Empty; }

@djee-ms Could you give us some advice about how to correctly address this problem of updating the UI on Unity after receiving the message? Am I doing it wrong with the MainThreadQueue that you already have on the Unity Example?

NOTE: I just noticed that OnPostInitialize is already called from the Main Thread Queue so it's not necessary to do it again when declaring the listener for the MessageReceived function. So at the end I only use it on my ChatMessageReceived function like this:

_mainThreadWorkQueue.Enqueue(() => { textReceivedMesh.text = text; });

So it is no more necessary to use the temporal variable on the Update method.

[pablomarcos]

@pablomarcos > Fixed in fb4a230, can you try again with that commit please? I am pretty sure that was it.

And for this issue I can confirm you that now the messages are received on the HoloLens without any problem. Bi-directional communication now without problems. Thanks!

[djee-ms]

@pablomarcos that code looks good.

On the threading question, assigning on one thread a handler to the MessageReceived event does not give any guarantee on which thread the event will be fired. This is standard C#. In the case of the C# library, for performance reason the event is fired on whatever thread the underlying C++ callback is invoked, which is never the UI thread. If you have some extra requirements on threading, you need to handle that yourself, like you did by deferring the work to the UI thread. And yes the purpose of OnPostInitialize is to do work on the UI thread after the PeerConnection is initialized.

[astaikos316]

@pablomarcos, @djee-ms Thank you for all the help getting the data channel to work. I am now able to get messages flowing between apps.

[astaikos316]

@djee-ms I have been battling the next challenge here with the data channels. With the way the data channel receive is setup now in OnPostInitialize(), I cannot receive the meshes because unity throws this exception: UnityException: Internal_Create can only be called from the main thread, and hard crashes Unity.

[djee-ms]

General background

At the moment the events from MixedReality-WebRTC objects are not always invoked from the main UI thread. This is a decision that I may review and change, but this is currently like this to avoid the cost of asynchronous deferring when not needed. This means that many things cannot be done directly in the handler (delegate) you attach to the event. In particular, Unity requires that all calls to GameObject etc. be done exclusively on the main UI thread. So you have to defer your work to the Update() method for example.

Example

PeerConnection.cs has an example of doing this thing using its _mainThreadWorkQueue. For example, when the _nativePeer.InitializeAsync() call completes its task, the continuation is done on an undefined thread; at that point the code enqueue inside _mainThreadWorkQueue an action to invoke the OnPostInitialize() method.

https://github.com/microsoft/MixedReality-WebRTC/blob/1d4bd2fee5ebe0daddf0b1e2be772809fd057410/libs/Microsoft.MixedReality.WebRTC.Unity/Assets/Microsoft.MixedReality.WebRTC.Unity/Scripts/PeerConnection.cs#L442-L462

Then in its Update() method this action is dequeued and executed, therefore OnPostInitalize() is always called on the main UI thread and can safely access the Unity objects.

https://github.com/microsoft/MixedReality-WebRTC/blob/1d4bd2fee5ebe0daddf0b1e2be772809fd057410/libs/Microsoft.MixedReality.WebRTC.Unity/Assets/Microsoft.MixedReality.WebRTC.Unity/Scripts/PeerConnection.cs#L356-L363

Data channel messages

In general for other events you want to do the same if you access Unity objects. And remember that just because you register a handler for an event in a thread does not mean that your handler will be invoked in that thread. Registration and invoking are separate things. If your register a handler for the DataChannel.MessageReceived event, then that handler is likely to be called in an unspecified thread, so you cannot use Unity objects inside it. You need to use the same kind of work deferring as mentioned above.

I would do something like:

// This event may be fired from any thread, cannot use Unity objects directly
dataChannel.MessageReceived += (byte[] msg) => {
  // Enqueue an action for the main UI thread
  _mainThreadWorkQueue.Enqueue(() => {
    OnDataChannelMessageReceived(dataChannel, msg);
  });
});

Then copy the Update() method of PeerConnection.cs, and also write your handler from above:

/// Always called on main UI thread.
public OnDataChannelMessageReceived(DataChannel dataChannel, byte[] msg) {
  var go = new GameObject();
  // etc.
}

[pablomarcos]

Thanks for the great explanation Jerome. That’s very useful.

On mié, sept 25, 2019 a las 08:11, Jerome Humbert notifications@github.com escribió:

General background

At the moment the events from MixedReality-WebRTC objects are not always invoked from the main UI thread. This is a decision that I may review and change, but this is currently like this to avoid the cost of asynchronous deferring when not needed. This means that many things cannot be done directly in the handler (delegate) you attach to the event. In particular, Unity requires that all calls to GameObject etc. be done exclusively on the main UI thread. So you have to defer your work to the Update() method for example.

Example

PeerConnection.cs has an example of doing this thing using its _mainThreadWorkQueue. For example, when the _nativePeer.InitializeAsync() call completes its task, the continuation is done on an undefined thread; at that point the code enqueue inside _mainThreadWorkQueue an action to invoke the OnPostInitialize() method.

https://github.com/microsoft/MixedReality-WebRTC/blob/1d4bd2fee5ebe0daddf0b1e2be772809fd057410/libs/Microsoft.MixedReality.WebRTC.Unity/Assets/Microsoft.MixedReality.WebRTC.Unity/Scripts/PeerConnection.cs#L442-L462

Then in its Update() method this action is dequeued and executed, therefore OnPostInitalize() is always called on the main UI thread and can safely access the Unity objects.

https://github.com/microsoft/MixedReality-WebRTC/blob/1d4bd2fee5ebe0daddf0b1e2be772809fd057410/libs/Microsoft.MixedReality.WebRTC.Unity/Assets/Microsoft.MixedReality.WebRTC.Unity/Scripts/PeerConnection.cs#L356-L363

Data channel messages

In general for other events you want to do the same if you access Unity objects. And remember that just because you register a handler for an event in a thread does not mean that your handler will be invoked in that thread. Registration and invoking are separate things. If your register a handler for the DataChannel.MessageReceived event, then that handler is likely to be called in an unspecified thread, so you cannot use Unity objects inside it. You need to use the same kind of work deferring as mentioned above.

I would do something like:

//

This event may be fired from any thread, cannot use Unity objects directly

dataChannel

.

MessageReceived

+=

(

byte

[]

msg

)

=>

{

//

Enqueue an action for the main UI thread

_mainThreadWorkQueue

.

Enqueue

(()

=>

{

OnDataChannelMessageReceived

(

dataChannel

,

msg

); }); });

Then copy the Update() method of PeerConnection.cs, and also write your handler from above:

///

Always called on main UI thread.

public

OnDataChannelMessageReceived

(

DataChannel

dataChannel

,

byte

[]

msg

) {

var

go

=

new

GameObject

();

//

etc.

}

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[djee-ms]

@astaikos316 / @pablomarcos > Closing this one as I believe all questions were answered and there was no activity for quite a while. If there is anything else then please feel free to reopen.

[zxzkf1992]

@astaikos316 Sorry I was out of the office for several days. So following @djee-ms advice this is what I did... First thing is to add the new data channel we will use for sending our text messages (or wathever thing you want). I'm doing this on the PeerConnection.cs script, exactly on method OnPostInitialize() after the line OnInitialized.Invoke();

And this is the code I'm using for creating the data channel:

_nativePeer.AddDataChannelAsync(ChatChannelID, "chat", true, true).ContinueWith((prevTask) =>
{ 
    if (prevTask.Exception != null)
    {
        throw prevTask.Exception;
    }
    var newDataChannel = prevTask.Result;

    _mainThreadWorkQueue.Enqueue(() =>
    {
        _chatDataChannel = newDataChannel;
        _chatDataChannel.MessageReceived += ChatMessageReceived;
        Debug.Log("DataChannel for CHAT created successfully.");
    });
});

Note that ChatMessageReceived must run on the main thread so the first idea was to use the queue that already exists. The problem here is that after all it seems not to be running on the main thread so when I try to update my UI on my ChatMessageReceived function the app is crashing. So for solving this I have to be sure that I'm modifying the UI from the main thread. I did this storing the message received on a temporal variable and checking on the Update() method if this variable has a new value, then I write it on my UI.

private void ChatMessageReceived(byte[] message) { string text = System.Text.Encoding.UTF8.GetString(message); textToProcess = text; }

And as you can imagine on my Update function: if (textToProcess != String.Empty) { textReceivedMesh.text = textToProcess; textToProcess = String.Empty; }

@djee-ms Could you give us some advice about how to correctly address this problem of updating the UI on Unity after receiving the message? Am I doing it wrong with the MainThreadQueue that you already have on the Unity Example?

NOTE: I just noticed that OnPostInitialize is already called from the Main Thread Queue so it's not necessary to do it again when declaring the listener for the MessageReceived function. So at the end I only use it on my ChatMessageReceived function like this:

_mainThreadWorkQueue.Enqueue(() => { textReceivedMesh.text = text; });

So it is no more necessary to use the temporal variable on the Update method.

Thank you very much for sharing the code, which solved my problem ！