Foundation.Data from AVFrame’s `data` property

[Sajjon]

Im not great at pointer syntax, how can I convert an AVFrame instance data into Swift Foundation Data

(End goal is to convert a Bink Video to an AVPlayer playable item, Im trying each AVFrame => Data => CIImage => CVPixelBuffer => append to AVAssetWriteeInput (AVAssetWriterInputPixelBufferAdaptor))

[sunlubo]

video data is stored in AVFrame.data

[Sajjon]

Yes, that's what I said, and its type is UnsafeMutableBufferPointer<UnsafeMutablePointer<UInt8>?>

And Im asking how I can convert that type into Data?

[Sajjon]

@sunlubo Or rather, how can I create an CIImage or CGImage from an AVFrame? :)

[Sajjon]

Ah I found this: https://github.com/sunlubo/SwiftFFmpeg/issues/2

[Sajjon]

But it is outdated and still references the (now updated) and hard to use type data, hmm...

[gregcotten]

To initialize without copying (you must hold onto the AVFrame and make sure it's not unref'd until you're done with Data!

let data = Data(bytesNoCopy: .init(frame.buffer[0]!.data), count: Int(frame.buffer[0]!.size), deallocator: .none)

An untested method to have the Data safely deallocate the frame when no longer needed:

let data = Data(bytesNoCopy: .init(frame.buffer[0]!.data), count: Int(frame.buffer[0]!.size), deallocator: .custom({ _, bytes in
                        assert(bytes == Int(frame.buffer[0]!.size))
                        frame.unref()
                    }))

To initialize with Data by copying:

Data(buffer: .init(start: frame.buffer[0]!.data, count: Int(frame.buffer[0]!.size)))

Remember if you are sampling planar frame data (planar audio channels or YCbCr data), you'll need to get each valid plane (I was just using an index of 0)

[Sajjon]

@gregcotten thx a lot!

Edit; I got it working! Thx for the help. Managed convert a BINK Video to a Mov file.

Hmm im a little bit over my head here, I wanna play a BINK (bik files) video on macOS and Im able to read frame data using (Swift)FFmpeg. And I thought I would be able to save each individual frame as an CIImage and write usinf AVAsserWriter and that way somehow save the whole series of frames as a .MOV or similar movie that I can play with AVPlayer.

I've tried this, but CIImage initializer fails.

var dataSingleFrame = Data()
for (frameDataIndex, maybeFrameData) in frame.data.enumerated() {
    guard let frameData = maybeFrameData else {
        break
    }
    let byteCount = Int(frame.linesize[frameDataIndex])
    let singleFramePartData = Data(bytesNoCopy: .init(frameData), count: byteCount, deallocator: .none)
    dataSingleFrame.append(contentsOf: singleFramePartData)
}

guard let ciImage = CIImage(data: dataSingleFrame) else {
    fatalError("Failed to create CI image from data")
}

Any idea?

[gregcotten]

@Sajjon Apologies, I've amended my response to use AVFrame's buffer array instead.

[Sajjon]

Thx for the help @gregcotten ! maybe we ought to Swiftify those properties and change to [Data] instead of UnsafeMutableBufferPointer<UnsafeMutablePointer<UInt8>?> ?

[chris-mds]

Hi @Sajjon ,

did you have a working code snippet of your function? I modified it like the following code, but still no successful conversion to the image property. Even better would be a conversion function to CMSampleBuffer Maybe some ideas?

var dataSingleFrame = Data()
for (_, maybeFrameData) in frame.buf.enumerated() {
  guard let frameData = maybeFrameData else {
      break
  }
  let singleFramePartData = Data(bytesNoCopy: .init(frameData.data), count: frameData.size, deallocator: .none)
  dataSingleFrame.append(contentsOf: singleFramePartData)
}

guard let ciImage = CIImage(data: dataSingleFrame) else {
  print("Failed to create CI image from data")
  continue
}