Not able to utilize kCVPixelFormatType_32ARGB for custom encoder/CustomVideoSource

[theprojectabot]

In the CustomVideoSource sample it constructs a frame of kCVPixelFormatType_24RGB and submits to the encoder. Here I am constructing a frame from a UIImage that is of type kCVPixelFormatType_32ARGB

It seems I am able to successfully 'stream' to the publisher but when viewing a web client there is not visible frame. Just white. Looking at red5pro server logs it does look like its 'working' - Im under the impression that perhaps the R5VideoSource doesnt support kCVPixelFormatType_32ARGB?

var result = CVPixelBufferCreateWithBytes(kCFAllocatorDefault,
                                                      (image?.width)!,
                                                      (image?.height)!,
                                                      kCVPixelFormatType_32ARGB,
                                                      UnsafeMutableRawPointer( mutating: baseAddress!),
                                                      (image?.bytesPerRow)!,
                                                      { releaseContext, baseAddress in
                                                                                                    let contextData = Unmanaged<CFData>.fromOpaque(releaseContext!)
                                                                                                    contextData.release()
            },
                                                      unmanagedData.toOpaque(),
                                                      nil,
                                                      &red5pixelBuffer)

            if(result != kCVReturnSuccess){
                NSLog("Failed to get pixel buffer");
            }

            var videoInfo: CMVideoFormatDescription?;

            //Create a description for the pixel buffer
            result = CMVideoFormatDescriptionCreateForImageBuffer(kCFAllocatorDefault, red5pixelBuffer!, &videoInfo);

            if(result != kCVReturnSuccess) {
                NSLog("Failed to create video info");
            }

            //Only PTS is needed for the encoder - leave everything else invalid if you want
            var timingInfo: CMSampleTimingInfo = kCMTimingInfoInvalid;
            timingInfo.duration = kCMTimeInvalid;
            timingInfo.decodeTimeStamp = kCMTimeInvalid;
            timingInfo.presentationTimeStamp = self.PTS;

            var buffer: CMSampleBuffer?;

            //Create the sample buffer for the pixel buffer
            result = CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault,
                                                        red5pixelBuffer!,
                                                        true, nil, nil,
                                                        videoInfo!,
                                                        &timingInfo,
                                                        &buffer);

            //push the sample buffer to the encoder with type r5_media_type_video_custom
            //if(!self.pauseEncoding){
            self.encoder.encodeFrame( buffer, of: r5_media_type_video_custom );
            //}

[theprojectabot]

or kCVPixelFormatType_32BGRA for that matter.

[HoldenMills]

h264 is ingesting a 12-bit image. Red5 Pro converts from 24 to 12 internally as a convenience for the end user h264 does not support transparency, so internally we have to skip bytes on the 32-bit image to read it as a 24-bit image. You could convert it to 24bits manually by skipping the 4th byte or using lib functions if those are available.

-Holden

[theprojectabot]

@HoldenMills thanks for the feedback. So it is required to use 24rgb - Ill need to use the Accelerate .framework to do a fast conversion.

or are you saying that its ok for me to do kCVPixelFormatType_32ARGB because red5 will skip the bytes on the 32 bit image?

[mrchrisallen]

You will need to convert it 24rgb. The alfa channel will not work with Red5 Pro.

[theprojectabot]

@mrchrisallen do you have a spec with the exact format that is requried for the r5_media_type_video_custom? I want to make sure my frame sizes, colorspace and timing info are correct. I can duplicate what is in the CustomVideoSource sample but that is at: width: 352, height: 288 kCVPixelFormatType_24RGB with a timing source with 15fps. Id like to drive the timer from a CADisplayLink and increment the PTA off that.

My use case is Im doing a full screen UIView capture so depending on some devices I will be auto scaling down to a fixed size for the frame - but it will always maintain a specific aspect ratio.

Thoughts? Do you have any code that properly formats a UIImage to the specifics needed for r5_media_type_video_custom encode pixel buffer?

[mrchrisallen]

We don't have a spec on this, but I will track down more info now that we understand your use case. Stay tuned!

[theprojectabot]

@mrchrisallen got it working :), one trick other then the pixel buffer stuff I had to do was... audio is required otherwise the video wont render...Dont know why this is. And btw the audio is very latent, like 2 secs compared to the video stream. Ill make a new bug on that.

[mrchrisallen]

Okay excellent! It might actually be helpful to get an Xcode project that reproduces the issues you are seeing (using your custom video source). We can then debug it ourselves over here.

[theprojectabot]

@mrchrisallen Ill work on getting you a sample to send privately.

[theprojectabot]

@mrchrisallen emailed sample proj (info in the readme I supplied) to @HoldenMills

[mrchrisallen]

@theprojectabot Okay, fantastic! Thanks for doing that. We will check it out and get back to you.

[karam28]

@theprojectabot Hi, as you have successfully 'stream' to the publisher through CustomVideoSource sample. Can you please help me to do that. I am unable to understand how I can do that. If you can send me one sample, how i can do that. I will be very thankful to you. Please help me.