search

Unity-Technologies / UnityRenderStreaming

Streaming server for Unity
Other
1.31k stars 365 forks source link

[BUG]: iOS doesn't like the profile ID sent with the video codecs from the hardware encoder #677

Closed doctorpangloss closed 2 years ago

doctorpangloss commented 2 years ago

Package version

3.1.0-exp.3

Environment

* OS: Windows 11
* Unity version: Unity 2021.2.15f1
* Graphics API: DirectX 11
* Browser: Safari 15 / Mobile Safari 15

Steps To Reproduce

With software encoder (2.4.0-exp5):

Offer:

[Log] RTCPeerConnection::setRemoteDescription(DF098618) Setting remote description to:
v=0
o=- 7816953026285853028 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1
a=extmap-allow-mixed
a=msid-semantic: WMS 9261ef0d-0fc2-4b70-b59c-64ad4e22ed1e
m=video 9 UDP/TLS/RTP/SAVPF 127 120 125 119 124 118 35 36 123 117 122
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:UMj6
a=ice-pwd:dZKBU/r/i7PlFEWKKfDt3A4a
a=ice-options:trickle
a=fingerprint:sha-256 47:CE:BF:ED:B1:50:64:91:8D:F2:A2:B3:C9:51:4B:5B:D8:95:DF:C5:F8:AC:D7:70:95:60:CD:4E:3F:89:D3:8E
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 urn:3gpp:video-orientation
a=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:9261ef0d-0fc2-4b70-b59c-64ad4e22ed1e d3df7868-c3bf-4082-812c-c7ac22d2c6b2
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:127 VP8/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 transport-cc
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=rtpmap:120 rtx/90000
a=fmtp:120 apt=127
a=rtpmap:125 VP9/90000
a=rtcp-fb:125 goog-remb
a=rtcp-fb:125 transport-cc
a=rtcp-fb:125 ccm fir
a=rtcp-fb:125 nack
a=rtcp-fb:125 nack pli
a=fmtp:125 profile-id=0
a=rtpmap:119 rtx/90000
a=fmtp:119 apt=125
a=rtpmap:124 VP9/90000
a=rtcp-fb:124 goog-remb
a=rtcp-fb:124 transport-cc
a=rtcp-fb:124 ccm fir
a=rtcp-fb:124 nack
a=rtcp-fb:124 nack pli
a=fmtp:124 profile-id=2
a=rtpmap:118 rtx/90000
a=fmtp:118 apt=124
a=rtpmap:35 AV1X/90000
a=rtcp-fb:35 goog-remb
a=rtcp-fb:35 transport-cc
a=rtcp-fb:35 ccm fir
a=rtcp-fb:35 nack
a=rtcp-fb:35 nack pli
a=rtpmap:36 rtx/90000
a=fmtp:36 apt=35
a=rtpmap:123 red/90000
a=rtpmap:117 rtx/90000
a=fmtp:117 apt=123
a=rtpmap:122 ulpfec/90000
a=ssrc-group:FID 1464642629 2543620536
a=ssrc:1464642629 cname:q8xoSMYyTot2uBja
a=ssrc:1464642629 msid:9261ef0d-0fc2-4b70-b59c-64ad4e22ed1e d3df7868-c3bf-4082-812c-c7ac22d2c6b2
a=ssrc:1464642629 mslabel:9261ef0d-0fc2-4b70-b59c-64ad4e22ed1e
a=ssrc:1464642629 label:d3df7868-c3bf-4082-812c-c7ac22d2c6b2
a=ssrc:2543620536 cname:q8xoSMYyTot2uBja
a=ssrc:2543620536 msid:9261ef0d-0fc2-4b70-b59c-64ad4e22ed1e d3df7868-c3bf-4082-812c-c7ac22d2c6b2
a=ssrc:2543620536 mslabel:9261ef0d-0fc2-4b70-b59c-64ad4e22ed1e
a=ssrc:2543620536 label:d3df7868-c3bf-4082-812c-c7ac22d2c6b2
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:UMj6
a=ice-pwd:dZKBU/r/i7PlFEWKKfDt3A4a
a=ice-options:trickle
a=fingerprint:sha-256 47:CE:BF:ED:B1:50:64:91:8D:F2:A2:B3:C9:51:4B:5B:D8:95:DF:C5:F8:AC:D7:70:95:60:CD:4E:3F:89:D3:8E
a=setup:actpass
a=mid:1
a=sctp-port:5000
a=max-message-size:262144

Answer:

[Log] PeerConnectionBackend::createAnswerSucceeded(DF098618) Create answer succeeded:
v=0
o=- 7165924746929344616 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1
a=extmap-allow-mixed
a=msid-semantic: WMS
m=video 9 UDP/TLS/RTP/SAVPF 127 120 125 119 123 117 122
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:ZkzO
a=ice-pwd:R9RMfh3unCPlbVaIpK3/dzwe
a=ice-options:trickle
a=fingerprint:sha-256 29:FB:85:73:53:00:C0:1F:ED:C7:6D:5B:8D:1F:3D:9F:BA:34:1A:2E:E1:9A:13:5E:86:36:35:37:36:AA:D5:59
a=setup:active
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 urn:3gpp:video-orientation
a=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:127 VP8/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 transport-cc
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=rtpmap:120 rtx/90000
a=fmtp:120 apt=127
a=rtpmap:125 VP9/90000
a=rtcp-fb:125 goog-remb
a=rtcp-fb:125 transport-cc
a=rtcp-fb:125 ccm fir
a=rtcp-fb:125 nack
a=rtcp-fb:125 nack pli
a=fmtp:125 profile-id=0
a=rtpmap:119 rtx/90000
a=fmtp:119 apt=125
a=rtpmap:123 red/90000
a=rtpmap:117 rtx/90000
a=fmtp:117 apt=123
a=rtpmap:122 ulpfec/90000
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:ZkzO
a=ice-pwd:R9RMfh3unCPlbVaIpK3/dzwe
a=ice-options:trickle
a=fingerprint:sha-256 29:FB:85:73:53:00:C0:1F:ED:C7:6D:5B:8D:1F:3D:9F:BA:34:1A:2E:E1:9A:13:5E:86:36:35:37:36:AA:D5:59
a=setup:active
a=mid:1
a=sctp-port:5000
a=max-message-size:262144

When only H264 profiles are enabled, Safari generates the following answer:

Offer:

[Log] RTCPeerConnection::setRemoteDescription(1FE10CA8) Setting remote description to:
v=0
o=- 9178090369549855783 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1
a=extmap-allow-mixed
a=msid-semantic: WMS
m=video 9 UDP/TLS/RTP/SAVPF 102 104 106
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:ZvlD
a=ice-pwd:ltOs00IN7Fc+0Xx3z0yyjBCa
a=ice-options:trickle
a=fingerprint:sha-256 8B:06:B0:9E:3F:F1:70:34:57:6E:29:31:F9:ED:9D:E4:3C:EB:97:CC:5B:98:53:97:A3:F6:F1:E8:FA:7D:21:B0
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 urn:3gpp:video-orientation
a=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendonly
a=msid:- 175b50ae-a21d-478a-9ad7-524e1a56559c
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:102 H264/90000
a=rtcp-fb:102 goog-remb
a=rtcp-fb:102 transport-cc
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack
a=rtcp-fb:102 nack pli
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=420033
a=rtpmap:104 H264/90000
a=rtcp-fb:104 goog-remb
a=rtcp-fb:104 transport-cc
a=rtcp-fb:104 ccm fir
a=rtcp-fb:104 nack
a=rtcp-fb:104 nack pli
a=fmtp:104 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640033
a=rtpmap:106 H264/90000
a=rtcp-fb:106 goog-remb
a=rtcp-fb:106 transport-cc
a=rtcp-fb:106 ccm fir
a=rtcp-fb:106 nack
a=rtcp-fb:106 nack pli
a=fmtp:106 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d0033
a=ssrc:1637622795 cname:pXkZtSAm5LKI3hu1
a=ssrc:1637622795 msid:- 175b50ae-a21d-478a-9ad7-524e1a56559c
a=ssrc:1637622795 mslabel:-
a=ssrc:1637622795 label:175b50ae-a21d-478a-9ad7-524e1a56559c
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:ZvlD
a=ice-pwd:ltOs00IN7Fc+0Xx3z0yyjBCa
a=ice-options:trickle
a=fingerprint:sha-256 8B:06:B0:9E:3F:F1:70:34:57:6E:29:31:F9:ED:9D:E4:3C:EB:97:CC:5B:98:53:97:A3:F6:F1:E8:FA:7D:21:B0
a=setup:actpass
a=mid:1
a=sctp-port:5000
a=max-message-size:262144

Answer:

[Log] PeerConnectionBackend::createAnswerSucceeded(1FE10CA8) Create answer succeeded:
v=0
o=- 5039973119986111046 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1
a=extmap-allow-mixed
a=msid-semantic: WMS
m=video 9 UDP/TLS/RTP/SAVPF 0
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:59WH
a=ice-pwd:HQfNMsMCBhnW2Ywb0ubuqQlB
a=ice-options:trickle
a=fingerprint:sha-256 E0:F2:CC:2B:D2:E5:4A:2A:D9:8D:D5:B8:F6:85:B1:62:8D:2C:2C:C5:31:46:92:96:0D:8B:18:7D:C1:6A:83:3D
a=setup:active
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 urn:3gpp:video-orientation
a=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=recvonly
a=rtcp-mux
a=rtcp-rsize
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:59WH
a=ice-pwd:HQfNMsMCBhnW2Ywb0ubuqQlB
a=ice-options:trickle
a=fingerprint:sha-256 E0:F2:CC:2B:D2:E5:4A:2A:D9:8D:D5:B8:F6:85:B1:62:8D:2C:2C:C5:31:46:92:96:0D:8B:18:7D:C1:6A:83:3D
a=setup:active
a=mid:1
a=sctp-port:5000
a=max-message-size:262144

Current Behavior

Safari fails to establish a connection.

Expected Behavior

It should connect.

Anything else?

Munging the SDP to say 42e01f instead of 420033 fixes the issue.

I also have to force the level to NV_ENC_LEVEL_5_2 on the encoder because that code also misinterprets the level and causes a crash.

doctorpangloss commented 2 years ago

Sorry this should be in com.unity.webrtc