search

microsoft / MixedReality-WebRTC

MixedReality-WebRTC is a collection of components to help mixed reality app developers integrate audio and video real-time communication into their application and improve their collaborative experience
https://microsoft.github.io/MixedReality-WebRTC/
MIT License
910 stars 284 forks source link

Unity Editor Crashes when receiving serialized Image through DataChannel #590

Open Orinion opened 4 years ago

Orinion commented 4 years ago

Describe the bug When sending an serialized object that contains a byte array that contains a raw image, the unity editor of the receiver crashes. The following error can be found in C:\Users\xyzl\AppData\Local\Unity\Editor\Editor.txt

(there are lines start with a hashtag but i removed it for better formatting) Fatal error in: ../../media/sctp/sctptransport.cc, line 301 last system error: 0 Check failed: transport->partial_message_.size() == 0 || rcv.rcv_sid == transport->partial_message_sid_ t be found. When moving or deleting files outside of Unity, please ensure that the corresponding .meta file is moved or deleted along with it.

I wrote a custom class to send serialized objects thru a webrtc dataChannel. The class works flawlessly for many objects such as as a byte array representing a Mesh. However when trying to send the imageBuffer the receiver crashes. My guess is that the package is too big (3686400 byte = 3.6 Mb)

To Reproduce Steps to reproduce the behavior:

  1. take picture on a hololens 1 using the PhotoCapture object and save it using photoCaptureFrame.CopyRawImageDataIntoBuffer(imageBuffer); (the resulting byte array has a size of 3686400)
  2. serialize object using a Binaryformatter and send it thru a dataChannel using dataChannel.SendMessage(ms.ToArray());
  3. try receiving it on a different application from a different application
  4. See error

Expected behavior The editor to not crash and instead receive the byte array or atleast give an errormessage like "message to long"

SDP messages SDP messages are extremely helpful to diagnose a wide range of issues from connection not being established to audio/video issues and codecs issues.

Offer message type=Offer data=v=0 o=- 6423240329577917595 3 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE 0 1 2 a=msid-semantic: WMS m=audio 64704 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126 c=IN IP4 131.234.120.235 a=rtcp:9 IN IP4 0.0.0.0 a=candidate:799006805 1 udp 2122260223 192.168.2.103 64704 typ host generation 0 network-id 1 network-cost 10 a=candidate:1629232293 1 tcp 1518280447 192.168.2.103 52441 typ host tcptype passive generation 0 network-id 1 network-cost 10 a=candidate:2933384929 1 udp 1686052607 131.234.120.235 64704 typ srflx raddr 192.168.2.103 rport 64704 generation 0 network-id 1 network-cost 10 a=ice-ufrag:CJch a=ice-pwd:jwVGLeddmXxpxe7rxjJgODre a=ice-options:trickle a=fingerprint:sha-256 3D:36:98:9F:AA:31:0A:8B:17:1D:92:F6:F2:D0:05:84:E9:76:F6:22:55:59:F7:82:2B:2D:72:68:81:D2:3C:83 a=setup:actpass a=mid:0 a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid a=sendrecv a=msid:- 6606fc61-5baa-4336-93f4-57f32237a18f a=rtcp-mux a=rtpmap:111 opus/48000/2 a=rtcp-fb:111 transport-cc a=fmtp:111 minptime=10;useinbandfec=1 a=rtpmap:103 ISAC/16000 a=rtpmap:104 ISAC/32000 a=rtpmap:9 G722/8000 a=rtpmap:102 ILBC/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:106 CN/32000 a=rtpmap:105 CN/16000 a=rtpmap:13 CN/8000 a=rtpmap:110 telephone-event/48000 a=rtpmap:112 telephone-event/32000 a=rtpmap:113 telephone-event/16000 a=rtpmap:126 telephone-event/8000 a=ssrc:4251833021 cname:OujSRx+C/hq0l4QF a=ssrc:4251833021 msid: 6606fc61-5baa-4336-93f4-57f32237a18f a=ssrc:4251833021 mslabel: a=ssrc:4251833021 label:6606fc61-5baa-4336-93f4-57f32237a18f m=video 64705 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127 124 125 c=IN IP4 131.234.120.235 a=rtcp:9 IN IP4 0.0.0.0 a=candidate:799006805 1 udp 2122260223 192.168.2.103 64705 typ host generation 0 network-id 1 network-cost 10 a=candidate:1629232293 1 tcp 1518280447 192.168.2.103 52442 typ host tcptype passive generation 0 network-id 1 network-cost 10 a=candidate:2933384929 1 udp 1686052607 131.234.120.235 64705 typ srflx raddr 192.168.2.103 rport 64705 generation 0 network-id 1 network-cost 10 a=ice-ufrag:CJch a=ice-pwd:jwVGLeddmXxpxe7rxjJgODre a=ice-options:trickle a=fingerprint:sha-256 3D:36:98:9F:AA:31:0A:8B:17:1D:92:F6:F2:D0:05:84:E9:76:F6:22:55:59:F7:82:2B:2D:72:68:81:D2:3C:83 a=setup:actpass a=mid:1 a=extmap:2 urn:ietf:params:rtp-hdrext:toffset a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:4 urn:3gpp:video-orientation a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07 a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid a=sendonly a=msid:- 48102110-7569-4a2c-847e-1ecf149cd96d a=rtcp-mux a=rtcp-rsize a=rtpmap:96 VP8/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtpmap:97 rtx/90000 a=fmtp:97 apt=96 a=rtpmap:98 VP9/90000 a=rtcp-fb:98 goog-remb a=rtcp-fb:98 transport-cc a=rtcp-fb:98 ccm fir a=rtcp-fb:98 nack a=rtcp-fb:98 nack pli a=fmtp:98 x-google-profile-id=0 a=rtpmap:99 rtx/90000 a=fmtp:99 apt=98 a=rtpmap:100 H264/90000 a=rtcp-fb:100 goog-remb a=rtcp-fb:100 transport-cc a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=fmtp:100 packetization-mode=1 a=rtpmap:101 rtx/90000 a=fmtp:101 apt=100 a=rtpmap:127 red/90000 a=rtpmap:124 rtx/90000 a=fmtp:124 apt=127 a=rtpmap:125 ulpfec/90000 a=ssrc-group:FID 1065929777 1563207672 a=ssrc:1065929777 cname:OujSRx+C/hq0l4QF a=ssrc:1065929777 msid: 48102110-7569-4a2c-847e-1ecf149cd96d a=ssrc:1065929777 mslabel: a=ssrc:1065929777 label:48102110-7569-4a2c-847e-1ecf149cd96d a=ssrc:1563207672 cname:OujSRx+C/hq0l4QF a=ssrc:1563207672 msid: 48102110-7569-4a2c-847e-1ecf149cd96d a=ssrc:1563207672 mslabel: a=ssrc:1563207672 label:48102110-7569-4a2c-847e-1ecf149cd96d m=application 64706 DTLS/SCTP 5000 c=IN IP4 131.234.120.235 a=candidate:799006805 1 udp 2122260223 192.168.2.103 64706 typ host generation 0 network-id 1 network-cost 10 a=candidate:1629232293 1 tcp 1518280447 192.168.2.103 52443 typ host tcptype passive generation 0 network-id 1 network-cost 10 a=candidate:2933384929 1 udp 1686052607 131.234.120.235 64706 typ srflx raddr 192.168.2.103 rport 64706 generation 0 network-id 1 network-cost 10 a=ice-ufrag:CJch a=ice-pwd:jwVGLeddmXxpxe7rxjJgODre a=ice-options:trickle a=fingerprint:sha-256 3D:36:98:9F:AA:31:0A:8B:17:1D:92:F6:F2:D0:05:84:E9:76:F6:22:55:59:F7:82:2B:2D:72:68:81:D2:3C:83 a=setup:actpass a=mid:2 a=sctpmap:5000 webrtc-datachannel 1024
Answer message type=Answer data=v=0 o=- 113560989482810982 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE 0 1 2 a=msid-semantic: WMS m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:brnG a=ice-pwd:xAiMw5SMcuhAdA7w5NCm4h40 a=ice-options:trickle a=fingerprint:sha-256 1F:D7:CC:A1:CD:51:44:E4:B7:DB:76:4D:81:AA:8C:D1:11:67:D0:9E:E8:EE:2B:DF:E9:EC:1D:22:45:81:1E:8C a=setup:active a=mid:0 a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid a=sendrecv a=msid:- 5b6ffcbf-8ed8-48cc-aed7-18605e13b6cb a=rtcp-mux a=rtpmap:111 opus/48000/2 a=rtcp-fb:111 transport-cc a=fmtp:111 minptime=10;useinbandfec=1 a=rtpmap:103 ISAC/16000 a=rtpmap:104 ISAC/32000 a=rtpmap:9 G722/8000 a=rtpmap:102 ILBC/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:106 CN/32000 a=rtpmap:105 CN/16000 a=rtpmap:13 CN/8000 a=rtpmap:110 telephone-event/48000 a=rtpmap:112 telephone-event/32000 a=rtpmap:113 telephone-event/16000 a=rtpmap:126 telephone-event/8000 a=ssrc:3190434653 cname:CxqDwx88nZPLONP7 m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 127 124 125 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:brnG a=ice-pwd:xAiMw5SMcuhAdA7w5NCm4h40 a=ice-options:trickle a=fingerprint:sha-256 1F:D7:CC:A1:CD:51:44:E4:B7:DB:76:4D:81:AA:8C:D1:11:67:D0:9E:E8:EE:2B:DF:E9:EC:1D:22:45:81:1E:8C a=setup:active a=mid:1 a=extmap:2 urn:ietf:params:rtp-hdrext:toffset a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:4 urn:3gpp:video-orientation a=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07 a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid a=recvonly a=rtcp-mux a=rtcp-rsize a=rtpmap:96 VP8/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtpmap:97 rtx/90000 a=fmtp:97 apt=96 a=rtpmap:98 VP9/90000 a=rtcp-fb:98 goog-remb a=rtcp-fb:98 transport-cc a=rtcp-fb:98 ccm fir a=rtcp-fb:98 nack a=rtcp-fb:98 nack pli a=fmtp:98 x-google-profile-id=0 a=rtpmap:99 rtx/90000 a=fmtp:99 apt=98 a=rtpmap:127 red/90000 a=rtpmap:124 rtx/90000 a=fmtp:124 apt=127 a=rtpmap:125 ulpfec/90000 m=application 9 DTLS/SCTP 5000 c=IN IP4 0.0.0.0 b=AS:30 a=ice-ufrag:brnG a=ice-pwd:xAiMw5SMcuhAdA7w5NCm4h40 a=ice-options:trickle a=fingerprint:sha-256 1F:D7:CC:A1:CD:51:44:E4:B7:DB:76:4D:81:AA:8C:D1:11:67:D0:9E:E8:EE:2B:DF:E9:EC:1D:22:45:81:1E:8C a=setup:active a=mid:2 a=sctpmap:5000 webrtc-datachannel 1024

Environment sender:

Additional context i know this is a complex problem but i hope i gave all information necessary. I am pretty certain that the problem is not on my side because i tried a lot to fix it. I don't think giving you all classes is needed, but let me know if i can provide more code/data.

The only think i can think of is that the byte array is too big (byte[] of size 3686400). When i try to send the byte[] {1} it doesn't crash. thanks for your time.

astaikos316 commented 4 years ago

My suggestion it to packetize the byte array before sending. I have found it more efficient and effective when trying to send large files.

fibann commented 4 years ago

+1, there is probably some internal buffer in the WebRTC implementation that cannot handle all that data in one message. It shouldn't crash arguably, OTOH I wouldn't expect this to work in general. So your best bet is to split the array and sending it bit by bit.

djee-ms commented 4 years ago

...and monitor the BufferingChanged event, so that you don't enqueue too much data too fast, otherwise the data channel will close itself internally.