search

twilio / video-quickstart-ios

Twilio Video Quickstart for iOS
https://www.twilio.com/docs/api/video
MIT License
457 stars 177 forks source link

ReplayKit & Audio #466

Open rcodarini opened 4 years ago

rcodarini commented 4 years ago

Description

I'm working to an iOS ReplayKit Broadcast Extension in order to stream my iPad screen to a Twilio Room. The video streaming is working fine but the audio doesn't work.

Steps to Reproduce

  1. I am in a room and the streaming is active
  2. I added an RPSystemBroadcastPickerView to start the Broadcast extension
  3. when the starts broadcasting I disconnect the app from the room and I connect the Broadcast Extension.
  4. Video streaming is working fine but Audio doesn't work. (In both directions)
  5. the function processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) is called without sampleBufferType == .audioMic

Logs

Debug level logs are helpful when investigating issues. To enable debug level logging, add the following code to your application:

Here the logs from the extension

2020-03-26 14:54:02.810070+0100 BroadcastExtension[2878:533155] [User Defaults] Couldn't read values in CFPrefsPlistSource<0x283729480> (Domain: group.it.space1.VirtualCollaborationExtensions, User: kCFPreferencesAnyUser, ByHost: Yes, Container: (null), Contents Need Refresh: Yes): Using kCFPreferencesAnyUser with a container is only allowed for System Containers, detaching from cfprefsd
2020-03-26 14:54:02.813657+0100 BroadcastExtension[2878:533155] INFO:Twilio:[Core](0x16af7b000): Creating peer connection signaling thread ...
2020-03-26 14:54:02.813838+0100 BroadcastExtension[2878:533155] INFO:Twilio:[Core](0x16af7b000): Creating peer connection worker thread ...
2020-03-26 14:54:02.813887+0100 BroadcastExtension[2878:533155] INFO:Twilio:[Core](0x16af7b000): Creating peer connection networking thread ...
2020-03-26 14:54:02.814443+0100 BroadcastExtension[2878:533155] INFO:Twilio:[Core](0x16af7b000): Creating peer connection factory ...
2020-03-26 14:54:02.820256+0100 BroadcastExtension[2878:533155] INFO:Twilio:[Core](0x16af7b000): Done creating peer connection factory.
2020-03-26 14:54:02.820333+0100 BroadcastExtension[2878:533155] DEBUG:Twilio:[Core](0x16af7b000): Done creating AudioMediaFactoryImpl.
2020-03-26 14:54:02.820495+0100 BroadcastExtension[2878:533155] INFO:Twilio:[Core](0x16af7b000): Creating a video track ...
2020-03-26 14:54:02.821793+0100 BroadcastExtension[2878:533155] INFO:Twilio:[Core](0x16af7b000): Creating a data track ...
2020-03-26 14:54:02.821991+0100 BroadcastExtension[2878:533155] INFO:Twilio:[Core](0x16af7b000): Adding audio track ...
2020-03-26 14:54:02.822548+0100 BroadcastExtension[2878:533155] DEBUG:Twilio:[Core](0x16af7b000): Creating dedicated notifier queue ...
2020-03-26 14:54:02.822712+0100 BroadcastExtension[2878:533155] DEBUG:Twilio:[Core](0x16af7b000): RoomSignalingImpl::RoomSignalingImpl()
2020-03-26 14:54:02.822800+0100 BroadcastExtension[2878:533155] DEBUG:Twilio:[Core](0x16af7b000): Opening connection cd03b20d-2ff1-4bd8-a1ac-c1f85df02d7c to wss://global.vss.twilio.com/signaling.
2020-03-26 14:54:02.822898+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Initialize connection cd03b20d-2ff1-4bd8-a1ac-c1f85df02d7c.
2020-03-26 14:54:02.823029+0100 BroadcastExtension[2878:533155] DEBUG:Twilio:[Core](0x16af7b000): Setting notifier queue. notifier_queue=0x281923c78
broadcastStartedWithSetupInfo:  Optional([:])
2020-03-26 14:54:02.839162+0100 BroadcastExtension[2878:533164] INFO:Twilio:[Core](0x16b2c3000): Will connect to host global.vss.twilio.com.
2020-03-26 14:54:02.857151+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Resolved host global.vss.twilio.com.
2020-03-26 14:54:02.859459+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Connecting to 52.59.103.57...
2020-03-26 14:54:03.581198+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Connected to 52.59.103.57.
2020-03-26 14:54:03.581395+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Established a TCP connection with 52.59.103.57.
2020-03-26 14:54:03.895591+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Completed a TLS handshake with 52.59.103.57.
2020-03-26 14:54:03.897952+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Sending WebSocket handshake:
GET /signaling HTTP/1.1
Host: global.vss.twilio.com
Upgrade: websocket
Connection: upgrade
Sec-WebSocket-Key: 3NOtYG1NIo/VN+G9En3oTQ==
Sec-WebSocket-Version: 13
Sec-WebSocket-Extensions: permessage-deflate; server_max_window_bits=15; client_max_window_bits=15
User-Agent: Boost.Beast/277 twilio-video-cpp/5.1.1
2020-03-26 14:54:03.947695+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Received WebSocket response:
HTTP/1.1 101 Switching Protocols
Date: Thu, 26 Mar 2020 13:54:03 GMT
Connection: upgrade
Sec-WebSocket-Accept: 6i1a/1rr7kDhOt+gN7176Lv6910=
Sec-WebSocket-Extensions: permessage-deflate
Upgrade: WebSocket
2020-03-26 14:54:03.947869+0100 BroadcastExtension[2878:533164] INFO:Twilio:[Core](0x16b2c3000): Completed WebSocket handshake with 52.59.103.57.
2020-03-26 14:54:03.948121+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Sending message (76 bytes):
{"id":"cd03b20d-2ff1-4bd8-a1ac-c1f85df02d7c","timeout":5000,"type":"hello"}
2020-03-26 14:54:03.996584+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Received message:
{"negotiatedTimeout":5000,"type":"welcome"}
2020-03-26 14:54:03.997313+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Connection cd03b20d-2ff1-4bd8-a1ac-c1f85df02d7c is ready.
2020-03-26 14:54:04.118490+0100 BroadcastExtension[2878:533154] DEBUG:Twilio:[Core](0x16aeef000): Parsing 'video' endpoint config: {"video":{"network_traversal_service":{"ttl":14400,"date_created":"Thu, 26 Mar 2020 13:54:04 +0000","date_updated":"Thu, 26 Mar 2020 13:54:04 +0000","capability_token":"video","ice_servers":[{"urls":"turn:global.turn.twilio.com:3478?transport=udp","username":"60a7f31d50a0cc4ad00bb70d154452f59bbd5d93bf3ab384d9ed3f25fec54369","credential":"O1cQa5HSJqkbUfex4Ck7cyjk8Nkfm2T2MRwgO84VJVk="},{"urls":"turns:global.turn.twilio.com:443?transport=tcp","username":"60a7f31d50a0cc4ad00bb70d154452f59bbd5d93bf3ab384d9ed3f25fec54369","credential":"O1cQa5HSJqkbUfex4Ck7cyjk8Nkfm2T2MRwgO84VJVk="}]}}}
2020-03-26 14:54:04.118653+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): RoomSignalingImpl: State transition successful: kInit -> kConnecting
2020-03-26 14:54:04.118708+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): AppleReachability::AppleReachability()
2020-03-26 14:54:04.118740+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): Creating zeroAddrReachability
2020-03-26 14:54:04.132764+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): AppleReachability::onConnectionEstablished, socket_fd = 16
2020-03-26 14:54:04.145462+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): AppleReachability started listening on address pair on 16 socket
2020-03-26 14:54:04.146170+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): Starting the Network Manager
2020-03-26 14:54:04.146295+0100 BroadcastExtension[2878:533161] INFO:Twilio:[Core](MediaFactoryImpl::signaling): Creating peer connection ...
2020-03-26 14:54:04.151670+0100 BroadcastExtension[2878:533161] INFO:Twilio:[Core](MediaFactoryImpl::signaling): Adding local stream to peer connection ...
2020-03-26 14:54:04.159912+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): addDataChannel() - Invoking on the signaling thread.
2020-03-26 14:54:04.160538+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): Open -> Updating. Process an event
2020-03-26 14:54:04.160597+0100 BroadcastExtension[2878:533161] DEBUG:Twilio:[Core](MediaFactoryImpl::signaling): Create local offer: AC8012d7AEeCBf268Ce1228E9B8F****
2020-03-26 14:54:04.162421+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): onCreateSessionLocalDescription AC8012d7AEeCBf268Ce1228E9B8F****
2020-03-26 14:54:04.163280+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): SDP Utils: New SSRC group 1870324610 4266260129 --- Original SSRC group 1870324610 4266260129
2020-03-26 14:54:04.163931+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): PeerConnection state: Updating -> Waiting
2020-03-26 14:54:04.164038+0100 BroadcastExtension[2878:533156] INFO:Twilio:[Core](0x16b007000): Local offer is ready for AC8012d7AEeCBf268Ce1228E9B8F****.
2020-03-26 14:54:04.164114+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): Queue Description: 1 for PeerConnection: AC8012d7AEeCBf268Ce1228E9B8F****.
2020-03-26 14:54:04.164913+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Sending message (5228 bytes):
{"body":{"format":"planb","ice_servers":"success","media_signaling":null,"participant":{"revision":1,"tracks":[{"enabled":true,"id":"54cDAbDada064feCCF10Ef6aFA2067D8","kind":"audio","name":"54cDAbDada064feCCF10Ef6aFA2067D8","priority":"standard"},{"enabled":true,"id":"686DC5dDfcD5Feff848B6DE2eeCBAB6d","kind":"video","name":"Screen","priority":"standard"},{"enabled":true,"id":"9eaAFfEb329BD33F1D1CB5dE5D09D2eB","kind":"data","name":"Screencast_DataTrack","priority":"standard"}]},"peer_connections":[{"description":{"revision":1,"sdp":"v=0\r\no=- 8367677294399528823 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE audio video data\r\na=msid-semantic: WMS DD4b183B15F4aAdDf1b358c4bb33162A\r\nm=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:9880\r\na=ice-pwd:xW/Apg/XuK8hFZzoIrZBCBYo\r\na=ice-options:trickle\r\na=fingerprint:sha-256 67:C4:51:C5:77:A0:E6:E3:53:78:F4:05:E1:2B:42:2B:E4:F4:7C:92:EC:1C:D6:5C:89:62:C5:C3:F4:C8:DC:C7\r\na=setup:actpass\r\na=mid:audio\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=sendrecv\r\na=rtcp-mux\r\na=rtpmap:111 opus/48000/2\r\na=rtcp-fb:111 transport-cc\r\na=fmtp:111 minptime=10;useinbandfec=1\r\na=rtpmap:103 ISAC/16000\r\na=rtpmap:104 ISAC/32000\r\na=rtpmap:9 G722/8000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:106 CN/32000\r\na=rtpmap:105 CN/16000\r\na=rtpmap:13 CN/8000\r\na=rtpmap:110 telephone-event/48000\r\na=rtpmap:112 telephone-event/32000\r\na=rtpmap:113 telephone-event/16000\r\na=rtpmap:126 telephone-event/8000\r\na=ssrc:2001103989 cname:AupfQthmRaR8fVpo\r\na=ssrc:2001103989 msid:DD4b183B15F4aAdDf1b358c4bb33162A 54cDAbDada064feCCF10Ef6aFA2067D8\r\na=ssrc:2001103989 mslabel:DD4b183B15F4aAdDf1b358c4bb33162A\r\na=ssrc:2001103989 label:54cDAbDada064feCCF10Ef6aFA2067D8\r\nm=video 9 UDP/TLS/RTP/SAVPF 100 96 97 98 99 101 102 125 127\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:9880\r\na=ice-pwd:xW/Apg/XuK8hFZzoIrZBCBYo\r\na=ice-options:trickle\r\na=fingerprint:sha-256 67:C4:51:C5:77:A0:E6:E3:53:78:F4:05:E1:2B:42:2B:E4:F4:7C:92:EC:1C:D6:5C:89:62:C5:C3:F4:C8:DC:C7\r\na=setup:actpass\r\na=mid:video\r\na=extmap:2 urn:ietf:params:rtp-hdrext:toffset\r\na=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:4 urn:3gpp:video-orientation\r\na=extmap:5 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay\r\na=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type\r\na=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/video-timing\r\na=sendrecv\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:96 VP8/90000\r\na=rtcp-fb:96 goog-remb\r\na=rtcp-fb:96 transport-cc\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\na=rtpmap:97 rtx/90000\r\na=fmtp:97 apt=96\r\na=rtpmap:98 VP9/90000\r\na=rtcp-fb:98 goog-remb\r\na=rtcp-fb:98 transport-cc\r\na=rtcp-fb:98 ccm fir\r\na=rtcp-fb:98 nack\r\na=rtcp-fb:98 nack pli\r\na=rtpmap:99 rtx/90000\r\na=fmtp:99 apt=98\r\na=rtpmap:100 H264/90000\r\na=rtcp-fb:100 goog-remb\r\na=rtcp-fb:100 transport-cc\r\na=rtcp-fb:100 ccm fir\r\na=rtcp-fb:100 nack\r\na=rtcp-fb:100 nack pli\r\na=fmtp:100 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f\r\na=rtpmap:101 rtx/90000\r\na=fmtp:101 apt=100\r\na=rtpmap:102 red/90000\r\na=rtpmap:125 rtx/90000\r\na=fmtp:125 apt=102\r\na=rtpmap:127 ulpfec/90000\r\na=ssrc-group:FID 1870324610 4266260129\r\na=ssrc:1870324610 cname:AupfQthmRaR8fVpo\r\na=ssrc:1870324610 msid:DD4b183B15F4aAdDf1b358c4bb33162A 686DC5dDfcD5Feff848B6DE2eeCBAB6d\r\na=ssrc:1870324610 mslabel:DD4b183B15F4aAdDf1b358c4bb33162A\r\na=ssrc:1870324610 label:686DC5dDfcD5Feff848B6DE2eeCBAB6d\r\na=ssrc:4266260129 cname:AupfQthmRaR8fVpo\r\na=ssrc:4266260129 msid:DD4b183B15F4aAdDf1b358c4bb33162A 686DC5dDfcD5Feff848B6DE2eeCBAB6d\r\na=ssrc:4266260129 mslabel:DD4b183B15F4aAdDf1b358c4bb33162A\r\na=ssrc:4266260129 label:686DC5dDfcD5Feff848B6DE2eeCBAB6d\r\nm=application 9 DTLS/SCTP 5000\r\nc=IN IP4 0.0.0.0\r\na=ice-ufrag:9880\r\na=ice-pwd:xW/Apg/XuK8hFZzoIrZBCBYo\r\na=ice-options:trickle\r\na=fingerprint:sha-256 67:C4:51:C5:77:A0:E6:E3:53:78:F4:05:E1:2B:42:2B:E4:F4:7C:92:EC:1C:D6:5C:89:62:C5:C3:F4:C8:DC:C7\r\na=setup:actpass\r\na=mid:data\r\na=sctpmap:5000 webrtc-datachannel 1024\r\n","type":"offer"},"id":"AC8012d7AEeCBf268Ce1228E9B8F****"}],"publisher":{"hw_device_arch":"arm64","hw_device_manufacturer":"Apple","hw_device_model":"iPad7,4","name":"twilio-video-ios","platform_name":"iOS","platform_version":"13.3.1","sdk_version":"3.2.2"},"subscribe":{"revision":1,"rules":[{"all":true,"type":"exclude"}]},"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsImN0eSI6InR3aWxpby1mcGE7dj0xIn0.eyJpc3MiOiJTSzFmMzZhMThhNmRkMTJmNWEyMDA4ZDUyYzZmMzY1MGE4IiwiZXhwIjoxNTg1MjM0NDAwLCJqdGkiOiJTSzFmMzZhMThhNmRkMTJmNWEyMDA4ZDUyYzZmMzY1MGE4LTE1ODUyMzA4MDAiLCJzdWIiOiJBQ2I4NWIxYjRjYWZlNzUyNmI3NDA4M2ViMDc3NjNhMGQzIiwiZ3JhbnRzIjp7ImlkZW50aXR5IjoidGVzdHJvYkBzcGFjZTEuaXQiLCJ2aWRlbyI6eyJyb29tIjoiMGYwOTE3YzctNTcwNS00ZjFjLWIwM2EtMDI0MmYyMzYyMzg1In19fQ.rp_UtEfN4l1xz4Xf5sftjLQ2r-Fty28eDQBjMn9dOIU","type":"connect","version":2},"type":"msg"}
2020-03-26 14:54:04.603934+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Received message:
{"body":{"version":2,"type":"connected","peer_connections":[{"id":"AC8012d7AEeCBf268Ce1228E9B8F****","description":{"type":"answer","sdp":"v=0\r\no=- 3794219644 3794219644 IN IP4 0.0.0.0\r\ns=VM734a5bfbc3259b8322a201a50adc2e53\r\nt=0 0\r\na=ice-lite\r\na=msid-semantic: WMS *\r\na=group:BUNDLE audio video data\r\nm=audio 10861 UDP/TLS/RTP/SAVPF 111 0\r\nc=IN IP4 34.203.254.139\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=rtcp:10861 IN IP4 34.203.254.139\r\na=rtpmap:111 opus/48000/2\r\na=rtpmap:0 PCMU/8000\r\na=fmtp:111 minptime=10;useinbandfec=1\r\na=candidate:2 1 UDP 2013266430 34.203.254.139 10861 typ host\r\na=candidate:2 2 UDP 2013266429 34.203.254.139 17512 typ host\r\na=rtcp-mux\r\na=setup:active\r\na=mid:audio\r\na=recvonly\r\na=ice-ufrag:nTtH\r\na=ice-pwd:O1qAPFmJUTMBBeQBi5pRA5\r\na=fingerprint:sha-256 6E:B9:F5:77:F4:23:46:85:79:80:D4:0D:A9:B6:10:2F:27:74:7A:FD:8A:2B:4E:06:A2:37:63:D9:7B:C4:FA:F1\r\nm=video 10861 UDP/TLS/RTP/SAVPF 100 102 127\r\nc=IN IP4 34.203.254.139\r\na=rtcp-rsize\r\na=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=rtcp:10861 IN IP4 34.203.254.139\r\na=rtpmap:100 H264/90000\r\na=rtpmap:102 red/90000\r\na=rtpmap:127 ulpfec/90000\r\na=rtcp-fb:100 goog-remb\r\na=rtcp-fb:100 ccm fir\r\na=rtcp-fb:100 nack\r\na=rtcp-fb:100 nack pli\r\na=fmtp:100 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f\r\na=candidate:2 1 UDP 2013266430 34.203.254.139 10861 typ host\r\na=candidate:2 2 UDP 2013266429 34.203.254.139 17512 typ host\r\na=rtcp-mux\r\na=setup:active\r\na=mid:video\r\na=recvonly\r\na=ice-ufrag:nTtH\r\na=ice-pwd:O1qAPFmJUTMBBeQBi5pRA5\r\na=fingerprint:sha-256 6E:B9:F5:77:F4:23:46:85:79:80:D4:0D:A9:B6:10:2F:27:74:7A:FD:8A:2B:4E:06:A2:37:63:D9:7B:C4:FA:F1\r\nm=application 10861 DTLS/SCTP 5000\r\nc=IN IP4 34.203.254.139\r\na=sctpmap:5000 webrtc-datachannel 1024\r\na=candidate:2 1 UDP 2013266430 34.203.254.139 10861 typ host\r\na=candidate:2 2 UDP 2013266429 34.203.254.139 17512 typ host\r\na=mid:data\r\na=ice-ufrag:nTtH\r\na=ice-pwd:O1qAPFmJUTMBBeQBi5pRA5\r\na=fingerprint:sha-256 6E:B9:F5:77:F4:23:46:85:79:80:D4:0D:A9:B6:10:2F:27:74:7A:FD:8A:2B:4E:06:A2:37:63:D9:7B:C4:FA:F1\r\n","revision":1}}],"sid":"RMfd0dd2878041a67e8cafc58759b5fcbf","name":"0f0917c7-5705-4f1c-b03a-0242f2362385","participant":{"sid":"PAbbf370c7cd758478b0d94d48a58b1***","identity":"testrob@*****.***","tracks":[{"kind":"audio","priority":"standard","id":"54cDAbDada064feCCF10Ef6aFA2067D8","enabled":true,"sid":"MTf1d834ba5080ec0037ee4e4613ed1a51","name":"54cDAbDada064feCCF10Ef6aFA2067D8","state":"ready"},{"kind":"video","priority":"standard","id":"686DC5dDfcD5Feff848B6DE2eeCBAB6d","enabled":true,"sid":"MT90151e416345e2955b1c929cf503b9a7","name":"Screen","state":"ready"}],"revision":1,"state":"connected"},"participants":[{"sid":"PA163c3d844cce468bd691d223b32***","identity":"re@*****.***","tracks":[{"kind":"data","priority":"standard","id":"fb3da328-1fae-4246-a2ec-eda740d6d8ee","enabled":true,"sid":"MT52b1bf2fe7bc54e6fd35f01cefd0cea4","name":"1b6cd7ac-0fe0-434e-93e0-bc550d84edd6","state":"ready"}],"revision":5,"state":"connected"}],"recording":{"enabled":false,"revision":1},"subscribed":{"revision":0,"tracks":[]},"published":{"revision":1,"tracks":[{"kind":"audio","priority":"standard","id":"54cDAbDada064feCCF10Ef6aFA2067D8","enabled":true,"sid":"MTf1d834ba5080ec0037ee4e4613ed1a51","name":"54cDAbDada064feCCF10Ef6aFA2067D8","state":"ready"},{"kind":"data","priority":"standard","id":"9eaAFfEb329BD33F1D1CB5dE5D09D2eB","enabled":true,"name":"Screencast_DataTrack","state":"created"},{"kind":"video","priority":"standard","id":"686DC5dDfcD5Feff848B6DE2eeCBAB6d","enabled":true,"sid":"MT90151e416345e2955b1c929cf503b9a7","name":"Screen","state":"ready"}]},"session":"b85b1b4cafe7526b74083eb07763a0d3fd0dd2878041a67e8cafc58759b5fcbfbbf370c7cd758478b0d94d48a58b12dce39a75b77b6f3050efd7aeca6771e7a8","options":{"media_region":"us1","signaling_region":"de1","session_timeout":30}},"type":"msg"}
2020-03-26 14:54:04.604813+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Received message:
{"body":{"version":2,"type":"update","sid":"RMfd0dd2878041a67e8cafc58759b5fcbf","name":"0f0917c7-5705-4f1c-b03a-0242f2362385","participant":{"sid":"PAbbf370c7cd758478b0d94d48a58b1***","identity":"testrob@*****.***","tracks":[],"revision":1,"state":"connected"},"participants":[{"sid":"PA163c3d844cce468bd691d223b32***","identity":"re@*****.***","tracks":[{"kind":"data","priority":"standard","id":"fb3da328-1fae-4246-a2ec-eda740d6d8ee","enabled":true,"sid":"MT52b1bf2fe7bc54e6fd35f01cefd0cea4","name":"1b6cd7ac-0fe0-434e-93e0-bc550d84edd6","state":"ready"}],"revision":5,"state":"connected"}],"recording":{"enabled":false,"revision":1},"subscribe":{"revision":1,"rules":[{"type":"exclude","all":true}]},"published":{"revision":1,"tracks":[]}},"type":"msg"}
2020-03-26 14:54:04.604935+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): onAccepted
2020-03-26 14:54:04.604991+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): RoomSignalingImpl: State transition successful: kConnecting -> kConnected
2020-03-26 14:54:04.605166+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): PeerConnection state: Waiting -> Updating
2020-03-26 14:54:04.605235+0100 BroadcastExtension[2878:532900] DEBUG:Twilio:[Core](0x105235800): RemoteParticipantImpl::RemoteParticipantImpl: *** sid: PA163c3d844cce468bd691d****
2020-03-26 14:54:04.605220+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): Process remote answer at revision 1.
2020-03-26 14:54:04.605278+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): Process remote sdp for: AC8012d7AEeCBf268Ce1228E9B8F**** revision is: 1.
2020-03-26 14:54:04.605302+0100 BroadcastExtension[2878:532900] INFO:Twilio:[Core](0x105235800): Remote participant re@*****.*** added a data track with sid: MT52b1bf2fe7bc54e6fd35f01cefd0cea4, name: 1b6cd7ac-0fe0-434e-93e0-bc550d84edd6, enabled: 1
didConnectToRoom:  2020-03-26 14:54:04.606020+0100 BroadcastExtension[2878:533161] DEBUG:Twilio:[Core](MediaFactoryImpl::signaling): Applying local description to: AC8012d7AEeCBf268Ce1228E9B8F**** rev: 1
<<TVIRoom: 0x283222f40> name: 0f0917c7-5705-4f1c-b03a-0242f2362385, state: Connected, sid: RMfd0dd2878041a67e8cafc58759b5fcbf, delegate: <BroadcastExtension.SampleHandler: 0x282b20750>>
Remote participant PA163c3d844cce468bd691d223b32***
Remote participant PA163c3d844cce468bd691d223b32***
2020-03-26 14:54:04.610433+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): onSetSessionLocalDescription: AC8012d7AEeCBf268Ce1228E9B8F****
2020-03-26 14:54:04.610476+0100 BroadcastExtension[2878:533163] DEBUG:Twilio:[Core](MediaFactoryImpl::networking): void twilio::media::NetworkMonitor::onNetworksChanged()
2020-03-26 14:54:04.610546+0100 BroadcastExtension[2878:533163] DEBUG:Twilio:[Core](MediaFactoryImpl::networking): Network Name: en0, Cost: 10, Type : Wifi, Preference : 127, Active : 1, id: 1, prefix : 192.168.1.0 and key : en0%192.168.1.0/24
2020-03-26 14:54:04.610579+0100 BroadcastExtension[2878:533163] DEBUG:Twilio:[Core](MediaFactoryImpl::networking): Network Name: en2, Cost: 10, Type : Wifi, Preference : 126, Active : 1, id: 2, prefix : 169.254.0.0 and key : en2%169.254.0.0/16
2020-03-26 14:54:04.610608+0100 BroadcastExtension[2878:533163] DEBUG:Twilio:[Core](MediaFactoryImpl::networking): Network Name: pdp_ip0, Cost: 900, Type : Cellular, Preference : 125, Active : 1, id: 5, prefix : 10.40.78.74 and key : pdp_ip0%10.40.78.74/32
2020-03-26 14:54:04.610636+0100 BroadcastExtension[2878:533163] DEBUG:Twilio:[Core](MediaFactoryImpl::networking): Network Name: lo0, Cost: 0, Type : Loopback, Preference : 124, Active : 1, id: 4, prefix : ::1 and key : lo0%::1/128
2020-03-26 14:54:04.610667+0100 BroadcastExtension[2878:533163] DEBUG:Twilio:[Core](MediaFactoryImpl::networking): Network Name: lo0, Cost: 0, Type : Loopback, Preference : 123, Active : 1, id: 3, prefix : 127.0.0.0 and key : lo0%127.0.0.0/8
2020-03-26 14:54:04.610693+0100 BroadcastExtension[2878:533163] DEBUG:Twilio:[Core](MediaFactoryImpl::networking): New preferred network reported by manager: key: en0%192.168.1.0/24, type: Wifi, cost: 10, id: 1
2020-03-26 14:54:04.611130+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): Starting ICE Gathering timer...
2020-03-26 14:54:04.611226+0100 BroadcastExtension[2878:533161] DEBUG:Twilio:[Core](MediaFactoryImpl::signaling): Applying remote description to: AC8012d7AEeCBf268Ce1228E9B8F**** rev: 1
2020-03-26 14:54:04.614732+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): Done processing onSetSessionLocalDescription: AC8012d7AEeCBf268Ce1228E9B8F****
2020-03-26 14:54:04.614827+0100 BroadcastExtension[2878:533165] INFO:Twilio:[Core](0x16b34f000): Ice Gathering for AC8012d7AEeCBf268Ce1228E9B8F****.
2020-03-26 14:54:04.614861+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): Gathered a candidate. candidate:2222700650 1 udp 2122260223 192.168.1.105 54567 typ host generation 0 ufrag 9880 network-id 1 network-cost 10
2020-03-26 14:54:04.614893+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): Gathered a candidate. candidate:1464683403 1 udp 2122194687 169.254.98.236 62930 typ host generation 0 ufrag 9880 network-id 2 network-cost 10
2020-03-26 14:54:04.614921+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): Gathered a candidate. candidate:3430144856 1 udp 2122129151 10.40.78.74 49248 typ host generation 0 ufrag 9880 network-id 5 network-cost 900
2020-03-26 14:54:04.614949+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): ICE connection state transitioned from New -> Checking
2020-03-26 14:54:04.614978+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): PeerConnection AC8012d7AEeCBf268Ce1228E9B8F**** Ice connection state transitioned from New -> Checking
2020-03-26 14:54:04.615006+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): Room Media state changed to Checking
2020-03-26 14:54:04.615033+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): Ice Connection state checking. Stopping Ice gathering timer.
2020-03-26 14:54:04.615156+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): onSetSessionRemoteDescription: AC8012d7AEeCBf268Ce1228E9B8F****
2020-03-26 14:54:04.615250+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): Queue ICE candidate revision: 1 for PeerConnection: AC8012d7AEeCBf268Ce1228E9B8F****.
2020-03-26 14:54:04.615285+0100 BroadcastExtension[2878:533165] INFO:Twilio:[Core](0x16b34f000): Publish ICE candidate revision: 1 for PeerConnection: AC8012d7AEeCBf268Ce1228E9B8F****.
2020-03-26 14:54:04.615833+0100 BroadcastExtension[2878:533165] DEBUG:Twilio:[Core](0x16b34f000): Updating -> Open
2020-03-26 14:54:04.616423+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Sending message (733 bytes):
{"body":{"participant":{"revision":1,"tracks":[{"enabled":true,"id":"54cDAbDada064feCCF10Ef6aFA2067D8","kind":"audio","name":"54cDAbDada064feCCF10Ef6aFA2067D8","priority":"standard"},{"enabled":true,"id":"686DC5dDfcD5Feff848B6DE2eeCBAB6d","kind":"video","name":"Screen","priority":"standard"},{"enabled":true,"id":"9eaAFfEb329BD33F1D1CB5dE5D09D2eB","kind":"data","name":"Screencast_DataTrack","priority":"standard"}]},"peer_connections":[{"ice":{"candidates":[],"complete":true,"revision":1,"ufrag":"9880"},"id":"AC8012d7AEeCBf268Ce1228E9B8F****"}],"session":"b85b1b4cafe7526b74083eb07763a0d3fd0dd2878041a67e8cafc58759b5fcbfbbf370c7cd758478b0d94d48a58b12dce39a75b77b6f3050efd7aeca6771e7a8","type":"update","version":2},"type":"msg"}
2020-03-26 14:54:04.660185+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): Gathered a candidate. candidate:87277278 1 udp 1686052607 93.41.3.67 59766 typ srflx raddr 192.168.1.105 rport 54567 generation 0 ufrag 9880 network-id 1 network-cost 10
2020-03-26 14:54:04.806735+0100 BroadcastExtension[2878:533156] INFO:Twilio:[Core](0x16b007000): Ice Gathering Complete for AC8012d7AEeCBf268Ce1228E9B8F****.
2020-03-26 14:54:04.846564+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Received message:
{"body":{"version":2,"type":"update","sid":"RMfd0dd2878041a67e8cafc58759b5fcbf","name":"0f0917c7-5705-4f1c-b03a-0242f2362385","participant":{"sid":"PAbbf370c7cd758478b0d94d48a58b1***","identity":"testrob@*****.***","tracks":[{"kind":"audio","priority":"standard","id":"54cDAbDada064feCCF10Ef6aFA2067D8","enabled":true,"sid":"MTf1d834ba5080ec0037ee4e4613ed1a51","name":"54cDAbDada064feCCF10Ef6aFA2067D8","state":"ready"},{"kind":"video","priority":"standard","id":"686DC5dDfcD5Feff848B6DE2eeCBAB6d","enabled":true,"sid":"MT90151e416345e2955b1c929cf503b9a7","name":"Screen","state":"ready"}],"revision":1,"state":"connected"},"participants":[{"sid":"PA163c3d844cce468bd691d223b32***","identity":"re@*****.***","tracks":[{"kind":"data","priority":"standard","id":"fb3da328-1fae-4246-a2ec-eda740d6d8ee","enabled":true,"sid":"MT52b1bf2fe7bc54e6fd35f01cefd0cea4","name":"1b6cd7ac-0fe0-434e-93e0-bc550d84edd6","state":"ready"}],"revision":5,"state":"connected"}],"recording":{"enabled":false,"revision":1},"subscribed":{"revision":0,"tracks":[]},"published":{"revision":1,"tracks":[{"kind":"audio","priority":"standard","id":"54cDAbDada064feCCF10Ef6aFA2067D8","enabled":true,"sid":"MTf1d834ba5080ec0037ee4e4613ed1a51","name":"54cDAbDada064feCCF10Ef6aFA2067D8","state":"ready"},{"kind":"data","priority":"standard","id":"9eaAFfEb329BD33F1D1CB5dE5D09D2eB","enabled":true,"name":"Screencast_DataTrack","state":"created"},{"kind":"video","priority":"standard","id":"686DC5dDfcD5Feff848B6DE2eeCBAB6d","enabled":true,"sid":"MT90151e416345e2955b1c929cf503b9a7","name":"Screen","state":"ready"}]}},"type":"msg"}
Capture CMVideoDimensions(width: 1440, height: 1920) @ 16 fps.
Send CMVideoDimensions(width: 0, height: 0) @ 0 fps. RTT = 0 ms
2020-03-26 14:54:06.849787+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): ICE connection state transitioned from Checking -> Connected
2020-03-26 14:54:06.849869+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): PeerConnection AC8012d7AEeCBf268Ce1228E9B8F**** Ice connection state transitioned from Checking -> Connected
2020-03-26 14:54:06.849901+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): Room Media state changed to Connected
2020-03-26 14:54:06.849930+0100 BroadcastExtension[2878:533156] INFO:Twilio:[Core](0x16b007000): Media is connected
2020-03-26 14:54:06.850118+0100 BroadcastExtension[2878:533209] INFO:Twilio:[Core](0x16b797000): Connecting to sdkgw.us1.twilio.com:443.
2020-03-26 14:54:06.850361+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): ICE connection state transitioned from Connected -> Completed
2020-03-26 14:54:06.850431+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): PeerConnection AC8012d7AEeCBf268Ce1228E9B8F**** Ice connection state transitioned from Connected -> Completed
2020-03-26 14:54:06.850997+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): Room Media state changed to Completed
2020-03-26 14:54:06.856676+0100 BroadcastExtension[2878:533162] DEBUG:Twilio:[Platform](MediaFactoryImpl::worker): Render format did not changed or renderer is not available
2020-03-26 14:54:06.856802+0100 BroadcastExtension[2878:533162] INFO:Twilio:[Platform](MediaFactoryImpl::worker): Successfully started recording after format change.
2020-03-26 14:54:07.865234+0100 BroadcastExtension[2878:533209] INFO:Twilio:[Core](0x16b797000): Connected to sdkgw.us1.twilio.com:443.
2020-03-26 14:54:07.949804+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): Media for PeerConnection AC8012d7AEeCBf268Ce1228E9B8F**** transitioned from MediaInactive -> MediaActive
2020-03-26 14:54:07.949892+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): PeerConnection AC8012d7AEeCBf268Ce1228E9B8F**** Media activity state transitioned from MediaInactive -> MediaActive
2020-03-26 14:54:07.949925+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): Media Activity state changed to MediaActive
2020-03-26 14:54:07.949953+0100 BroadcastExtension[2878:533156] DEBUG:Twilio:[Core](0x16b007000): At least one media Track is active in Room.
Capture CMVideoDimensions(width: 1440, height: 1920) @ 15 fps.
Send CMVideoDimensions(width: 720, height: 540) @ 15 fps. RTT = 239 ms
2020-03-26 14:54:17.920703+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Received message:
{"body":{"version":2,"type":"update","sid":"RMfd0dd2878041a67e8cafc58759b5fcbf","name":"0f0917c7-5705-4f1c-b03a-0242f2362385","participant":{"sid":"PAbbf370c7cd758478b0d94d48a58b1***","identity":"testrob@*****.***","tracks":[{"kind":"audio","priority":"standard","id":"54cDAbDada064feCCF10Ef6aFA2067D8","enabled":true,"sid":"MTf1d834ba5080ec0037ee4e4613ed1a51","name":"54cDAbDada064feCCF10Ef6aFA2067D8","state":"ready"},{"kind":"data","priority":"standard","id":"9eaAFfEb329BD33F1D1CB5dE5D09D2eB","enabled":true,"sid":"MT644f31bef1938723e6b439f792ade***","name":"Screencast_DataTrack","state":"ready"},{"kind":"video","priority":"standard","id":"686DC5dDfcD5Feff848B6DE2eeCBAB6d","enabled":true,"sid":"MT90151e416345e2955b1c929cf503b9a7","name":"Screen","state":"ready"}],"revision":1,"state":"connected"},"participants":[{"sid":"PA163c3d844cce468bd691d223b32***","identity":"re@*****.***","tracks":[{"kind":"data","priority":"standard","id":"fb3da328-1fae-4246-a2ec-eda740d6d8ee","enabled":true,"sid":"MT52b1bf2fe7bc54e6fd35f01cefd0cea4","name":"1b6cd7ac-0fe0-434e-93e0-bc550d84edd6","state":"ready"}],"revision":5,"state":"connected"}],"recording":{"enabled":false,"revision":1},"subscribed":{"revision":0,"tracks":[]},"published":{"revision":2,"tracks":[{"kind":"audio","priority":"standard","id":"54cDAbDada064feCCF10Ef6aFA2067D8","enabled":true,"sid":"MTf1d834ba5080ec0037ee4e4613ed1a51","name":"54cDAbDada064feCCF10Ef6aFA2067D8","state":"ready"},{"kind":"data","priority":"standard","id":"9eaAFfEb329BD33F1D1CB5dE5D09D2eB","enabled":true,"sid":"MT644f31bef1938723e6b439f792ade***","name":"Screencast_DataTrack","state":"ready"},{"kind":"video","priority":"standard","id":"686DC5dDfcD5Feff848B6DE2eeCBAB6d","enabled":true,"sid":"MT90151e416345e2955b1c929cf503b9a7","name":"Screen","state":"ready"}]}},"type":"msg"}
2020-03-26 14:54:17.922476+0100 BroadcastExtension[2878:532900] DEBUG:Twilio:[Core](0x105235800): Dispatching onDataTrackPublished() for track sid: MT644f31bef1938723e6b439f792ade***
2020-03-26 14:54:19.025919+0100 BroadcastExtension[2878:533164] DEBUG:Twilio:[Core](0x16b2c3000): Sending message (21 bytes):

Versions

All relevant version information for the issue.

Video iOS SDK

3.2.2 via CocoaPods

Xcode

Xcode 11.4

iOS Version

13.3.1

iOS Device

iPad Pro 10,5"

ceaglest commented 4 years ago

Hi @rcodarini,

I'm working to an iOS ReplayKit Broadcast Extension in order to stream my iPad screen to a Twilio Room.

Are you trying to share just your application's screen or the entire device's screen (including the home screen and other apps)? A broadcast extension is not needed if you only want to share the application's screen.

Video streaming is working fine but Audio doesn't work. (In both directions)

A broadcast extension cannot playback audio. If you want that you will need to connect a Participant from your main application process.

the function processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) is called without sampleBufferType == .audioMic

It should be, did you enable the microphone when you tapped on the RPSystemBroadcastPickerView? If you don't tap to enable the microphone then audio samples of sampleBufferType == .audioMic will not be delivered to your extension.

I am working on some documentation improvements for our ReplayKit example which might help clarify the use cases. Until then, feel free to ask any more questions that you might have about how to use ReplayKit.

Best, Chris

rcodarini commented 4 years ago

Hi @ceaglest, sorry for the delay,

Are you trying to share just your application's screen or the entire device's screen (including the home screen and other apps)? A broadcast extension is not needed if you only want to share the application's screen.

I would like to share the entire screen, and it works fine

A broadcast extension cannot playback audio. If you want that you will need to connect a Participant from your main application process.

You mean that I have to leave the room connected in the main application? What happens when the app goes in background?

It should be, did you enable the microphone when you tapped on the RPSystemBroadcastPickerView? If you don't tap to enable the microphone then audio samples of sampleBufferType == .audioMic will not be delivered to your extension.

Yes, the microphone is enabled before I start broadcasting

I am working on some documentation improvements for our ReplayKit example which might help clarify the use cases. Until then, feel free to ask any more questions that you might have about how to use ReplayKit.

Great

Regards Roberto

dmichelutti commented 4 years ago

Hi @ceaglest I'm having the same issue described in this thread. I'm working on the Broadcast Extension to stream both video and audio. The main problem is that ReplayKit seems to reproduce only RPSampleBufferType.audioApp and RPSampleBufferType.video, but not RPSampleBufferType.audioMic, which is important for me to stream.

I've also followed this past thread: issue7. AndyBoyd seems to be in the same situation, but I don'understand what changes has he done on the example code to reach the goal. Also mixing two audio devices (audioApp and microphone) can be a good solution for me, but it seems doesn't work properly.

In SampleHandler file I have tried two ways: the first using your ReplayKitAudioCapturer file for audioDevice setting, second one using DefaultAudioDevice() and changing parameters with DefaultAudioDevice.DefaultAVAudioSessionConfigurationBlock(). In that block I have tried (no success) to change category and mode to AVAudioSession.sharedInstance().

I've noticed no errors in log, and the microphone is On, when I try to stream data. Have you some ideas of what I can do to stream also microphone audio? I have followed your ReplyKit Example code.

Thanks in advice,

David