Raw Data to Video via GPU

[MattFoley]

Considering that I have a raw stream of RGBA data per pixel that will be coming in one frame at a time, would GPUImage be an option for displaying that streaming data as video? If so, what part of GPUImage would I be looking at to do this?

[BradLarson]

Yes, you can do this with this framework. You'd be looking at the GPUImageRawDataInput class, which can take RGBA, BGRA, and RGB raw data as input. From there, it can be filtered, fed to the screen via a GPUImageView, or recorded to a movie.

[MattFoley]

Ah, Brad, next time you're in Chicago, I owe you a beer.

On Apr 6, 2013, at 12:26 PM, Brad Larson notifications@github.com wrote:

Yes, you can do this with this framework. You'd be looking at the GPUImageRawDataInput class, which can take RGBA, BGRA, and RGB raw data as input. From there, it can be filtered, fed to the screen via a GPUImageView, or recorded to a movie.

— Reply to this email directly or view it on GitHub.

[MattFoley]

I just replied to the StackOverflow post you helped me out on a year ago, and I just remembered I had this issue here as well. We were able to convert this raw data to video and it's been working for quite some time. I'm now working on recording data being passed into GPUImageRawDataInput and running into issues with the AVAssetWriter in GPUImageMovieWriter.

The error I'm getting is unfortunately completely opaque, my question is perhaps I've missed a few setup steps that were needed for recording from the GPUImageRawDataInput. I am successfully recording from a "GPUImageVideoCamera", but doing the exact same steps with the raw data input and a different file name is not working.

Are there extra steps needed I am missing?

This is an excerpt from my code that I think covers it: http://pastebin.com/LD6ZcJtN

Thanks for everything Brad!

[MattFoley]

Doing more research, I've found that the issue may be the video size that we're using. I'm getting the following console errors that weren't being piped into Xcode's Debugging Console: http://pastebin.com/bVQ7PJ12

It seems that the default codec being used, AVVideoCodecH264, is unable to handle 80x80 video size for recording. Do you know if it is possible to work around this in any way?

I appreciate any help you can give, thank you!

[BradLarson]

That’s a new one to me. I guess I haven’t tried recording a video smaller than 320x240, so I don’t know if that’s a hardware limitation or not. I honestly don’t do that much video recording, which kinda shows in the rough patches in the video input and outputs.

Have you tried changing the codec to the non-H.264 one (MPEG4 or Quicktime, I forget the constant for that)? That might possibly work where this fails.

On Jul 10, 2014, at 2:06 PM, MattFoley notifications@github.com wrote:

Doing more research, I've found that the issue may be the video size that we're using. I'm getting the following console errors that weren't being piped into Xcode's Debugging Console: http://pastebin.com/bVQ7PJ12

It seems that the default codec being used, AVVideoCodecH264, is unable to handle 80x80 video size for recording. Do you know if it is possible to work around this in any way?

I appreciate any help you can give, thank you!

— Reply to this email directly or view it on GitHub.

---

Brad Larson, Ph.D. Sunset Lake Software http://www.sunsetlakesoftware.com

[MattFoley]

Hmm, I understand, this is certainly a novel use case. Thanks for the incredibly quick response.

Looking at AVVideoSettings.h I only see "AVVideoCodecJPEG" as the other option, the others are OS X only. Trying that one gets around the initial error on "startWriting", but instead creates a different error on markAsFinished:

Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo=0x16db9840 {NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x16ebf1e0 "The operation couldn’t be completed. (OSStatus error -12142.)", NSLocalizedFailureReason=An unknown error occurred (-12142)}

The only change I made was to use these settings instead:

 NSDictionary *settings = @{ AVVideoCodecKey : AVVideoCodecJPEG,
                            AVVideoCompressionPropertiesKey : @{AVVideoQualityKey : @1.0},
                            AVVideoScalingModeKey : AVVideoScalingModeResizeAspect,
                            AVVideoWidthKey : @(kBufW),
                            AVVideoHeightKey : @(kBufH)};

Essentially what we're trying to do is use GPUImageMovieWriter to record our raw data which corresponds to per pixel values that are translated into colors by our lookup filter, thinking for playback we could use GPUImageRawDataOutput on a GPUImageMovieFile which would then pass the bytes back into a GPUImageRawDataInput and on into our custom lookup filter. Doing this we were thinking we'd be able to use AVPlayer to handle scrubbing, timing, etc.

This would be in lieu of creating our own pipeline for recording this raw data to file per frame and reading it back out, which is currently out of my depth.

Any advice you have would be very much appreciated, thanks!

[MattFoley]

Aha, oh Brad, I believe I found the problem. Or at least, a problem. We're using processData of GPUImageRawDataInput, which passes in kCMTimeInvalid to "newFrameReadyAtTime", so nothing is ever actually written into the file. This is likely the problem. Going to spend some time looking into setting up our own time tracking for frames.

[BradLarson]

Try using GPUImageRawDataInput's new -processDataForTimestamp:, which was added just for use in such a situation.