CVPixelBuffer to WebSocket

[yigitarik]

Hi everyone. I'm trying to send CVPixelBuffer to my websocket. But websocket has few data types for sending like "data, ping, pong, string". So when I try to convert that buffer data It returns empty. My goal is record the meeting in the server. At this point I could not send buffer data to my websocket. Can anyone help me? @stasel @kmadiar

[stasel]

Hey, sending Data through websockets should work. Can you share your code so others can get more context of what you are trying to achieve?

[yigitarik]

Hi again. Ok I am sharing my code.

This is my capturer code. In the end of this code block as you can see I added FrameRenderer() class to localVideTrack.

func startCaptureLocalVideo(renderer: RTCVideoRenderer) {

        guard let capturer = self.videoCapturer as? RTCCameraVideoCapturer else {
            return
        }

        guard
            let frontCamera = (RTCCameraVideoCapturer.captureDevices().first { $0.position == .front }),

            // choose highest res
            let format = (RTCCameraVideoCapturer.supportedFormats(for: frontCamera).sorted { (f1, f2) -> Bool in
                let width1 = CMVideoFormatDescriptionGetDimensions(f1.formatDescription).width
                let width2 = CMVideoFormatDescriptionGetDimensions(f2.formatDescription).width
                return width1 < width2
            }).last,

            // choose highest fps
            let fps = (format.videoSupportedFrameRateRanges.sorted { return $0.maxFrameRate < $1.maxFrameRate }.last) else {
            return
        }

        capturer.startCapture(with: frontCamera,
                              format: format,
                              fps: Int(fps.maxFrameRate))
        bufferDelegate?.capturer(capturer, didCapture: videoFrame!)
        localCapturer?.startCapture(with: frontCamera, format: format, fps: Int(fps.maxFrameRate))

        self.localVideoTrack?.add(renderer)

        self.localVideoTrack?.add(FrameRenderer())

    }

FrameRenderer() class is like:

class FrameRenderer: NSObject, RTCVideoRenderer {

    let manager = SocketManager(
        socketURL: URL(string: "http://socket.clumpapp.com")!,
        config: [.log(false), .compress, .secure(false), .selfSigned(true), .forcePolling(true) ])
    var socket:SocketIOClient!

//    private var signalClient: SignalingClient
    override init() {
        socket = manager.defaultSocket
        socket.connect()
        socket.on(clientEvent: .connect) {data, ack in
            print("Clump connected")
        }
        socket.on(clientEvent: .error) {data, ack in
            print("error")
        }
        super.init()
    }

    weak var delegate: FrameRendererDelegate?

    func setSize(_ size: CGSize) {
        print(size.width)
        print(size.height)
    }

    func renderFrame(_ frame: RTCVideoFrame?) {

        guard let rtcPixelBuffer = frame!.buffer as? RTCCVPixelBuffer else {
                print("Error: frame.buffer is not an RTCCVPixelBuffer")
                return
            }

            let imageBuffer = rtcPixelBuffer.pixelBuffer

            print(imageBuffer)

            var jsonString = convertPixelBufferToJSONData(pixelBuffer: imageBuffer)

            socket.emit("joinRoom", [jsonString])

    }
}

This socket is my example socket for testing. I am sending the data to joinRoom event.

And my convert function:

func convertPixelBufferToJSONData(pixelBuffer: CVPixelBuffer) -> Data? {
    var data = Data()
    data = Data()
    print("-*-*-*")
    CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.readOnly)
    let baseAddress = CVPixelBufferGetBaseAddress(pixelBuffer)
    CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.readOnly)

    let bufferSize = CVPixelBufferGetDataSize(pixelBuffer)
    data = Data(bytes: baseAddress!, count: bufferSize)
    print(baseAddress)
    print(data)
    print("-*-*-*")

    let jsonDictionary: [String: Any] = [
        "pixelBufferData": data.base64EncodedString()
    ]

    guard let jsonData = try? JSONSerialization.data(withJSONObject: jsonDictionary, options: []) else {
        print("nil")
        return nil
    }

    return jsonData
}

This convert function not working correctly I guess. As I said my sole purpose is recording the meeting on server simultaneously. I am open any other advice to do it. Thanks. @stasel