Closed MilanNosal closed 6 years ago
@MilanNosal
Can you tell me what version of TwilioVideo
you are using where you see this crash occurring?
Thanks,
Ryan
I used 2.3, yesterday I have updated to 2.4 and the crash still occurs.. Could it be something with renderers assigned to track? Or the room settings?
Thanks for getting back so quick. I don't believe it has anything to do with room settings. I just tested this in our internal application and I am not seeing the crash.
Is this crash occurring on the device where you are disabling the local video or on the remote participant side? Are you able to share the source showing how you are rendering the local video track?
I would like to try and get a setup where I can reproduce the crash you are seeing locally.
Thank you,
Ryan
@paynerc
Hi! So I've just found out that the same error happens in SampleHandler.swift
that's part of your example application when you try to disable the track there. it's from the ReplayKit example, there the broadcast extension. I believe you can replicate it very easily yourself, but here's the code with a couple of changes made by me to be able to try it out (to pinpoint my changes, I've added couple of FIXME: comments on places where I have added or changes something.. as you can see it's really almost no changes at all).
One thing that may be relevant is that even when you disable the local video track before connecting to room, it crashes only after the twilio connects to the room. Could it be the problem with room configuration, or with access token?
//
// SampleHandler.swift
// BroadcastExtension
//
// Created by Piyush Tank on 7/1/18.
// Copyright © 2018 Twilio. All rights reserved.
//
import Accelerate
import TwilioVideo
import ReplayKit
class SampleHandler: RPBroadcastSampleHandler, TVIRoomDelegate, TVIVideoCapturer {
public var isScreencast: Bool = true
// Video SDK components
public var room: TVIRoom?
weak var captureConsumer: TVIVideoCaptureConsumer?
static let kDesiredFrameRate = 30
static let kDownScaledFrameWidth = 540
static let kDownScaledFrameHeight = 960
let audioDevice = ExampleCoreAudioDevice()
// FIXME: Added reference so I get back to it later
var localScreenTrack: TVILocalVideoTrack?
public var supportedFormats: [TVIVideoFormat] {
get {
/*
* Describe the supported format.
* For this example we cheat and assume that we will be capturing the entire screen.
*/
let screenSize = UIScreen.main.bounds.size
let format = TVIVideoFormat()
format.pixelFormat = TVIPixelFormat.formatYUV420BiPlanarFullRange
format.frameRate = UInt(SampleHandler.kDesiredFrameRate)
format.dimensions = CMVideoDimensions(width: Int32(screenSize.width), height: Int32(screenSize.height))
return [format]
}
}
override func broadcastStarted(withSetupInfo setupInfo: [String : NSObject]?) {
TwilioVideo.audioDevice = ExampleCoreAudioDevice(audioCapturer: self)
// User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional.
// FIXME: My own access token
let accessToken = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsImN0eSI6InR3aWxpby1mcGE7dj0xIn0.eyJqdGkiOiJTSzJiNTgyZDc0ZDVlMDc2OTA0MmUxYzIxZTdiYWE1MWU5LTE1Mzg0MTg4NzkiLCJncmFudHMiOnsiaWRlbnRpdHkiOiIxNTIxMyIsInZpZGVvIjp7InJvb20iOiI2Rjk3OTM3Ri1EQzJCLTQ4MTQtQUI5Mi0wQjU2QzI3NDRCQkUifX0sImlhdCI6MTUzODQxODg3OSwiZXhwIjoxNTM4NDIyNDc5LCJpc3MiOiJTSzJiNTgyZDc0ZDVlMDc2OTA0MmUxYzIxZTdiYWE1MWU5Iiwic3ViIjoiQUNiYmJlNzZhODg1N2E0NzY0ZmZhZGUzODU2MDY1N2ZjZiJ9.Q6wcAVUisyJQRrGbQJ0qfU_LRS9qmLKyOZ__vVnRbZU";
// FIXME: storing reference to local track
localScreenTrack = TVILocalVideoTrack(capturer: self)
let h264VideoCodec = TVIH264Codec()
let localAudioTrack = TVILocalAudioTrack()
let connectOptions = TVIConnectOptions.init(token: accessToken) { (builder) in
// Use the local media that we prepared earlier.
builder.audioTracks = [localAudioTrack!]
builder.videoTracks = [self.localScreenTrack!]
// Use the preferred video codec
builder.preferredVideoCodecs = [h264VideoCodec] as! [TVIVideoCodec]
// The name of the Room where the Client will attempt to connect to. Please note that if you pass an empty
// Room `name`, the Client will create one for you. You can get the name or sid from any connected Room.
// FIXME: I commented out the if, since I'm using the same way in both cases + my own name of a room for which access token is valid
// if #available(iOS 12.0, *) {
builder.roomName = "6F97937F-DC2B-4814-AB92-0B56C2744BBE"
// } else {
// builder.roomName = setupInfo?["RoomName"] as? String
// }
}
// Connect to the Room using the options we provided.
room = TwilioVideo.connect(with: connectOptions, delegate: self)
// User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional.
print("broadcastStarted")
}
override func broadcastPaused() {
// User has requested to pause the broadcast. Samples will stop being delivered.
}
override func broadcastResumed() {
// User has requested to resume the broadcast. Samples delivery will resume.
}
override func broadcastFinished() {
// User has requested to finish the broadcast.
}
override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
RPScreenRecorder.shared().isMicrophoneEnabled = true
switch sampleBufferType {
case RPSampleBufferType.video:
if ((captureConsumer != nil) && room?.state == .connected) {
processVideoSampleBuffer(sampleBuffer)
}
break
case RPSampleBufferType.audioApp:
if (room?.state == .connected) {
ExampleCoreAudioDeviceRecordCallback(sampleBuffer)
}
break
case RPSampleBufferType.audioMic:
if (room?.state == .connected) {
ExampleCoreAudioDeviceRecordCallback(sampleBuffer)
}
break
}
}
var tempPixelBuffer : CVPixelBuffer?;
func startCapture(_ format: TVIVideoFormat, consumer: TVIVideoCaptureConsumer) {
captureConsumer = consumer
captureConsumer!.captureDidStart(true)
CVPixelBufferCreate(kCFAllocatorDefault,
480,//self.width,
640, //self.height,
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange,
nil,
&tempPixelBuffer)
}
func stopCapture() {
print("Stop capturing.")
}
// MARK:- Private
func processVideoSampleBuffer(_ sampleBuffer: CMSampleBuffer) {
// let imageBuffer = sampleBuffer.imageBuffer!
// let pixelBuffer = sampleBuffer.imageBuffer!
// FIXME: got error on previous line, had to replace it with this:
guard let pixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
var outPixelBuffer : CVPixelBuffer? = nil
CVPixelBufferLockBaseAddress(pixelBuffer, [])
let pixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer)
if (pixelFormat != kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) {
assertionFailure("Extension assumes the incoming frames are of type NV12")
}
let status = CVPixelBufferCreate(kCFAllocatorDefault,
SampleHandler.kDownScaledFrameWidth,
SampleHandler.kDownScaledFrameHeight,
pixelFormat,
nil,
&outPixelBuffer);
if (status != kCVReturnSuccess) {
print("Failed to create pixel buffer");
}
CVPixelBufferLockBaseAddress(outPixelBuffer!, []);
// Prepare source pointers.
var sourceImageY = vImage_Buffer(data: CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0),
height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(pixelBuffer, 0)),
width: vImagePixelCount(CVPixelBufferGetWidthOfPlane(pixelBuffer, 0)),
rowBytes: CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 0))
var sourceImageUV = vImage_Buffer(data: CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 1),
height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(pixelBuffer, 1)),
width:vImagePixelCount(CVPixelBufferGetWidthOfPlane(pixelBuffer, 1)),
rowBytes: CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 1))
// Prepare out pointers.
var outImageY = vImage_Buffer(data: CVPixelBufferGetBaseAddressOfPlane(outPixelBuffer!, 0),
height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(outPixelBuffer!, 0)),
width: vImagePixelCount(CVPixelBufferGetWidthOfPlane(outPixelBuffer!, 0)),
rowBytes: CVPixelBufferGetBytesPerRowOfPlane(outPixelBuffer!, 0))
var outImageUV = vImage_Buffer(data: CVPixelBufferGetBaseAddressOfPlane(outPixelBuffer!, 1),
height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(outPixelBuffer!, 1)),
width:vImagePixelCount( CVPixelBufferGetWidthOfPlane(outPixelBuffer!, 1)),
rowBytes: CVPixelBufferGetBytesPerRowOfPlane(outPixelBuffer!, 1))
var error = vImageScale_Planar8(&sourceImageY,
&outImageY,
nil,
vImage_Flags(0));
if (error != kvImageNoError) {
print("Failed to down scale luma plane ")
return;
}
error = vImageScale_CbCr8(&sourceImageUV,
&outImageUV,
nil,
vImage_Flags(0));
if (error != kvImageNoError) {
print("Failed to down scale chroma plane")
return;
}
CVPixelBufferUnlockBaseAddress(outPixelBuffer!, []);
CVPixelBufferUnlockBaseAddress(pixelBuffer, []);
let time: CMTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
let frame = TVIVideoFrame(timestamp: time,
buffer: outPixelBuffer!,
orientation: TVIVideoOrientation.up)
captureConsumer?.consumeCapturedFrame(frame!)
}
// MARK:- TVIRoomDelegate
func didConnect(to room: TVIRoom) {
print("didConnectToRoom")
// FIXME: Connect to the room and try to disable the track..
// When you disable it earlier, it still crashes only after connecting to the room
self.localScreenTrack?.isEnabled = false
}
func room(_ room: TVIRoom, didFailToConnectWithError error: Error) {
print("didFailToConnectWithError")
}
func room(_ room: TVIRoom, didDisconnectWithError error: Error?) {
print("didDisconnectWithError")
}
func room(_ room: TVIRoom, participantDidConnect participant: TVIRemoteParticipant) {
print("participantDidConnect")
}
func room(_ room: TVIRoom, participantDidDisconnect participant: TVIRemoteParticipant) {
print("participantDidDisconnect")
}
}
Having the same issue here, version 2.4.1. The other part of the conversation understands that the track is disabled.
This is a collapsed log to see some timestamps, first row is our own log that indicates that the track has been disabled.
2018-10-03 17:30:00.321818+0200 ***[12242:3536754] ***: [Twilio_Room_Local_Event] {"event":"disabledVideoTrack"}
2018-10-03 17:30:00.341165+0200 ***[12242:3537154] -[TVIRTCI420Buffer coreVideoFrameBuffer]: unrecognized selector sent to instance 0x281d58f40
I am digging into this right now and will post as soon as I am able to replicate.
Quick question, what iOS versions are you seeing this on?
Ryan
Both my devices are on iOS 12 GM
Great! In my case 12.0 also. Haven't tested it on 11.x.
OK. That's good to know. I will test on the various devices I have and see if I can reproduce and if there is any iOS version relation.
@MilanNosal, @aastlind
Also, what device models are you testing on?
iPhone 8 Plus. I’ll have a iPhone 6 with iOS 11.x available tomorrow, will test on that also.
OK. I have tested on both an iPhone 6Plus and an iPhone X with iOS 12 and am not seeing anything yet. I don't have an iPhone 8 Plus.. I have a 7 Plus with iOS 11 and was also not able to reproduce the issue.
A few more questions for you on this:
RoomSid
for a room that you were connected to when the crash occurred? I would like to pull the server logs to see the room configuration, etc. If you have a RoomSid
, you can email it to support@twilio.com and reference this ticket and then let me know that you sent it so I know to look for it.bt
in the debugger when the crash occurs and just attach that to this issue. Or if you are running outside of the debugger, the actual crash report would help a ton. I can resymbolicate that and hopefully get a better idea of what actually is happening.I am interested to hear if the crash is related to the iPhone 8 Plus or happens on other devices as well.
Ryan
Stack trace (bt):
* thread #72, queue = 'EncoderQueue', stop reason = signal SIGABRT
* frame #0: 0x000000018a509104 libsystem_kernel.dylib`__pthread_kill + 8
frame #1: 0x000000018a588a00 libsystem_pthread.dylib`pthread_kill$VARIANT$armv81 + 296
frame #2: 0x000000018a460d78 libsystem_c.dylib`abort + 140
frame #3: 0x0000000189b28f78 libc++abi.dylib`abort_message + 132
frame #4: 0x0000000189b29120 libc++abi.dylib`default_terminate_handler() + 304
frame #5: 0x0000000189b41e48 libobjc.A.dylib`_objc_terminate() + 124
frame #6: 0x0000000189b350fc libc++abi.dylib`std::__terminate(void (*)()) + 16
frame #7: 0x0000000189b34a40 libc++abi.dylib`__cxa_throw + 132
frame #8: 0x0000000189b41b84 libobjc.A.dylib`objc_exception_throw + 380
frame #9: 0x000000018a88b154 CoreFoundation`-[NSObject(NSObject) doesNotRecognizeSelector:] + 140
frame #10: 0x000000018a979810 CoreFoundation`___forwarding___ + 1412
frame #11: 0x000000018a97b4bc CoreFoundation`_CF_forwarding_prep_0 + 92
frame #12: 0x00000001057c354c TwilioVideo`-[TVIVideoEncoderH264 resetCompressionSessionIfNeededForPool:withFrame:] + 352
frame #13: 0x00000001057c2cac TwilioVideo`-[TVIVideoEncoderH264 encode:codecSpecificInfo:frameTypes:] + 224
frame #14: 0x0000000105800524 TwilioVideo`webrtc::(anonymous namespace)::ObjCVideoEncoder::Encode(webrtc::VideoFrame const&, webrtc::CodecSpecificInfo const*, std::__1::vector<webrtc::FrameType, std::__1::allocator<webrtc::FrameType> > const*) + 332
frame #15: 0x0000000105a1a358 TwilioVideo`webrtc::VCMGenericEncoder::Encode(webrtc::VideoFrame const&, webrtc::CodecSpecificInfo const*, std::__1::vector<webrtc::FrameType, std::__1::allocator<webrtc::FrameType> > const&) + 212
frame #16: 0x0000000105a2ad0c TwilioVideo`webrtc::vcm::VideoSender::AddVideoFrame(webrtc::VideoFrame const&, webrtc::CodecSpecificInfo const*) + 732
frame #17: 0x000000010583df58 TwilioVideo`webrtc::VideoStreamEncoder::EncodeVideoFrame(webrtc::VideoFrame const&, long long) + 1312
frame #18: 0x000000010583fd18 TwilioVideo`rtc::ClosureTask<webrtc::VideoStreamEncoder::OnFrame(webrtc::VideoFrame const&)::$_6>::Run() + 144
frame #19: 0x0000000105847f24 TwilioVideo`rtc::TaskQueue::Impl::TaskContext::RunTask(void*) + 60
frame #20: 0x000000010a954de4 libdispatch.dylib`_dispatch_client_callout + 16
frame #21: 0x000000010a95ce88 libdispatch.dylib`_dispatch_lane_serial_drain + 720
frame #22: 0x000000010a95db7c libdispatch.dylib`_dispatch_lane_invoke + 460
frame #23: 0x000000010a967c18 libdispatch.dylib`_dispatch_workloop_worker_thread + 1220
frame #24: 0x000000018a58e0f0 libsystem_pthread.dylib`_pthread_wqthread + 312
frame #25: 0x000000018a590d00 libsystem_pthread.dylib`start_wqthread + 4
I also emailed the RoomSid to support@twilio.com (support ticket 1528481).
See this on multiple devices
Hey @aastlind,
Thanks for the stack trace. I believe that, in 2.3.0 and later, we are making a bad assumption in the H.264 encoder about frame types, and formats. Disabling the Track causes software I420 buffers to be produced for the black frames, and our encoder was not expecting this type.
Regards, Chris
@ceaglest: Only to confirm, I just run a test removing H.264 as preferred video codec (leaving VP9 and VP8 as remaining preferred codecs) and then it works fine to disable the track.
Good to see that you know where to start! :)
@aastlind I apologize for the issue. Yes, we have a bug in our H.264 video encoder. I am working on the fix and aiming to release it in a couple of days. Thanks.
@MilanNosal @aastlind
We released Twilio Video iOS 2.5.1
today which addresses this crash.
Let us know if this has resolved your issue and then we'll close out this issue.
Thanks for your patience while we diagnosed and fixed the issue.
Ryan
@paynerc The issue is resolved, thank you for support and quick resolution.
HI @paynerc
I am getting this crash again with 'TwilioVideo', '~> 5' I am testing in iPad(6th Generation) iOS : 15.3.1
When I try to disable localVideoTrack, app just freeze in debug mode and on TestFlight build it crashes. I have that crash log if you want.
Also, I have sent an email of Sid
Thank you
@riddhixtraining,
Tagging @piyushtank on this issue as I am no longer with Twilio.
Ryan
I am using
TwilioVideo
for calls over the internet. I am trying to support disabling video.Based on the example code, I expected this would do the trick:
Where
localVideoTrack
is an instance ofTVILocalVideoTrack
created using:I used basically the same approach with the
TVILocalAudioTrack
to mute and unmute microphone, which works like a charm.However, with video, when I call:
I get a crash:
Any idea why this is happening?