ofVideoPlayer (AVFoundation-based) has poor performance with >1080 files

[mattfelsen]

Hi all. I've been doing some video tests with high resolution movie files (4K at 3840x2160 and 4096x2160 or 6K at 5760x3840) and I'm seeing very degraded performance in the current AVFoundation-based video player. Even running 1080 files is a little troublesome...the video will play at the correct speed of 24 or 30fps but the app will slow down to less than 60fps (seeing this on a 15"MBP).

For 4K, video plays about normally but the app will slow down to ~20-40fps. For 6K, the app & video are both about 1fps!

I've been running on @prisonerjohn's 64bit fork and I stopped tracking back in Nov. The video player at that point was the CLOUDS player. I was seeing 60fps for 1080 movies, and I do get app framerate drops (40-60fps) but still see smooth playback for both 4K 6K videos. This is both on a 15"MBP and Mac Pro.

I've been doing a number of comparisons to make sure I'm getting consistent behavior and to make sure I'm not misreporting this, but I see this with the following setups, both in 32bit and 64bit where applicable:

OF 0.8.4 with https://github.com/obviousjim/ofxAVFVideoPlayer or https://github.com/julapy/ofxAVFoundationVideoPlayer/tree/b6f891942f09b7fdd57862a2c9792166a7b67eca

OF dev before the switch in players (https://github.com/openframeworks/openFrameworks/tree/dee2fade3e24cbf37d3f03f51ce07cf9bf0d36ec) with native ofVideoPlayer or https://github.com/julapy/ofxAVFoundationVideoPlayer

OF dev after switch in players (https://github.com/openframeworks/openFrameworks/commit/f487707443147294e8c1b2e239bab77dee255d5d) with native ofVideoPlayer

(You'll have to revert some 0.9.0-style changes here and there to get these various combos to compile).

I haven't dug into the video playback code (nor do I think I have the expertise to really know what's going on here) but perhaps some of the playback code from the older player can be integrated with the current player and built on all of the great recent work from https://github.com/openframeworks/openFrameworks/pull/3736

[mattfelsen]

Forgot to add a ping to @openframeworks/video

[mattfelsen]

Also should add that I was testing with various ProRes codecs, mostly 422 LT and Proxy.

[davivid]

I'm seeing a problem with 3840 x 2160 h264 files - These used to play fine with 0.8.4 and the AVFoundation addon

[charlesveasey]

Not sure about the regression issue, but if using OSX and ProRes, in the file ofAVFoundationVideoPlayer.m try changing all (2) instances of the kCVPixelFormatType_32ARGB to kCVPixelFormatType_422YpCbCr8.

The video's getPixels() method will fail (unless updated), but the draw() or bind() methods seem ok.

I noticed much better performance with ProRes videos when switching to kCVPixelFormatType_422YpCbCr8.

AVFoundation lists optimized pixel formats here: https://developer.apple.com/library/mac/documentation/AVFoundation/Reference/AVAssetReaderTrackOutput_Class/index.html

[arturoc]

ofPixels already supports yuv so it should be relatively straight forward to implement getPixels() too. doing colorspace conversion is really cpu intensive so it's really possible that it's what causing the performance drop. ping @i-n-g-o

[harryhow]

I'm running into the similar performance issue with 0.8.4 + AVFoundation Addon. @mattfelsen I'm always seeing my fps low to 10-12 fps. My setup is : 1) Video: three videos @1080 x 3840px, h.264 codec, no audio, .mov format. size ~= 2~3GB (load three videos but only play one video at one time) 2) Output display/canvas: two 55" 4K TV with vertical extended screen setup (w x h = 2160 x 7680) 3) Mac Mini, 2.6GHz dual-core Intel Core i5, Intel Iris.

I don't do any getPixels or complex manipulation, only playing, pause, setDuration videos. Any idea to improve this low performance issue?

[ofTheo]

Could you try with 0.9.0? I would be curious to see if you get a similar fps.

Thanks Theo

[i-n-g-o]

i guess we would have similar performance with 0.9.0. proposal form charlesveasey sounds like the way to go, using kCVPixelFormatType_422YpCbCr8

maybe we should be able to specify which pixelformat to use when creating a player... i will have a look into that.

jep. colorspace conversion should not be done on cpu if possible. do we have some shaders for this? if we lack this, we might use parts of a benchmarking tool a colleague of mine made some time ago which also does colorspace conversion on the gpu. it's on github and open: https://github.com/ToolsOnAir/gl-frame-bender

[charlesveasey]

@harryhow i would use Pro Res not h.264 for larger videos if possible.

According to the docs: h.264 is CVPixelFormatType_420YpCbCr8BiPlanarVideoRange, or kCVPixelFormatType_420YpCbCr8BiPlanarFullRange pixel format

Yes, ideally the oF API allows one to set the pixel format, and any color conversion (if needed) is done on the GPU. Nice link @i-n-g-o

[arturoc]

@i-n-g-o we already have everything in place to do gpu colorspace conversion and specify the pixel format for the player.

the api works like:

video.setPixelFormat(OF_PIXELS_I420);

which would select yuv 420 or:

video.setPixelFormat(OF_PIXELS_NATIVE);

which would select the format the video is encoded with so there's no colorspace conversion at all in the cpu. then when you get new pixels just put them into an ofPixels specifying the pixel format when allocating it and the video player will do the right thing

this will only work with the programmable renderer

[i-n-g-o]

nice. i will have a look at it.

[harryhow]

@charlesveasey thanks for your suggestion, I will have a test with ProRes 422. Do I need to install pro res video codec in order to have better quality? https://support.apple.com/kb/dl1396?locale=en_US

[i-n-g-o]

native pixelFormat support work in progress: https://github.com/i-n-g-o/openFrameworks/tree/ofAVFoundationPlayer_set_pixelformats basic codecs are supprted: h264, 422YpCbCr8, ProRes (the rest needs some research and defaults to 422YpCbCr8 for now)

the pixelFormat is chosen from the codec-information obtained by CMFormatDescription (best guess). since some codecs support different chroma subsampling i'm not quite sure if this is the correct way to determine the pixelFormat. i did not find another way to determine the native video pixelFormat until now. any input on how to do that with AVFoundation/CoreMedia welcome ;)

with these changes the cpu-usage playing back a prores HD file drops from ~40% to ~25% on my machine (using kCVPixelFormatType_422YpCbCr8)

[mattfelsen]

@i-n-g-o That sounds good progress! I can test this with my old videos but it will be a few weeks before I'm back in front of a suitable machine. Do your updates also only work with the programmable pipeline, or also the fixed pipeline?

@harryhow Apple may have built ProRes decoding support into QuickTime Player, but I believe you will need to install the codecs to encode videos with ProRes. I usually use MPEG Streamclip to do H.264 -> ProRes encoding.

[i-n-g-o]

@mattfelsen the changes are independent of the gl-pipeline used. it's about setting a pixelFormat for the AVFoundation decoders.

the changes default to OF_PIXELS_NATIVE to use the best guessed pixelformat for a codec-type. a ofPixelFormat can be set with e.g.: movieplayer.setPixelFormat(OF_PIXELS_RGBA);

[i-n-g-o]

change of strategy... figuring out the internal pixelformat automatically with the codec information seems not reliable.

changed the branch like this: if the movieplayers pixelformat is set to OF_PIXELS_NATIVE (the default) it behaves like it used to be: using kCVPixelFormatType_32ARGB for osx and kCVPixelFormatType_32BGRA for ios, converting pixels in getPixels() to OF_PIXELS_RGBA. when setting a different pixelformat, it will fill ofPixels with the raw-decoded pixels of the frame (this may be yuv-pixels)

the user is responsible for setting a propper pixelformat. e.g.:

ofAVFoundationPlayer fingerMovie;

// OF_PIXELS_NATIVE
// ARGB 4:4:4:4 - uses kCVPixelFormatType_32ARGB internally
// converts pixels to OF_PIXELS_RGBA when fingerMovie.getPixels()
fingerMovie.setPixelFormat(OF_PIXELS_NATIVE);

// OF_PIXELS_UYVY
// YUV 4:2:2 - uses kCVPixelFormatType_422YpCbCr8 internally
// performance boost when using correct codec, returns plain yuv pixels with fingerMovie.getPixels()
fingerMovie.setPixelFormat(OF_PIXELS_UYVY);

// OF_PIXELS_I420
// YUV 4:2:0 - uses kCVPixelFormatType_420YpCbCr8BiPlanarFullRange internally
// likely to fail
fingerMovie.setPixelFormat(OF_PIXELS_I420);

// OF_PIXELS_RGBA
// RGBA 4:4:4:4 - uses kCVPixelFormatType_32RGBA internally
// this will fail
fingerMovie.setPixelFormat(OF_PIXELS_RGBA);

// introducing new ofPixelFormat: OF_PIXELS_ARGB
// ARGB 4:4:4:4 - uses kCVPixelFormatType_32ARGB internally
// performance improvement compared to OF_PIXELS_NATIVE when fingerMovie.getPixels(): no pixelconversion done on cpu
fingerMovie.setPixelFormat(OF_PIXELS_ARGB);

// load a movie
fingerMovie.load("movies/fingers.mov");

setPixelFormat() has to be called before loading a movie...

comments welcome ;)

[arturoc]

@i-n-g-o the idea about OF_PIXELS_NATIVE is that it should use the fastest format available and there should be no conversion at all, even if that format is yuv.

then when using the programable renderer, the renderer will do the colospace conversion using a shader

[bakercp]

If it's helpful, I recently implemented OF_PIXELS_NATIVE support for ofxPS3Grabber https://github.com/bakercp/ofxPS3EyeGrabber.

By default it uses RGB, which requires a colorspace conversion, but if you set it to OF_PIXELS_NATIVE it eliminates the pixel copy and renders the YUY2 pixels with the shader (if the programmable renderer is used).

e.g.

https://github.com/bakercp/ofxPS3EyeGrabber/blob/master/src/ofxPS3EyeGrabber.cpp#L299-L304

[bakercp]

Here's how it handles unsupported pixel types ...

https://github.com/bakercp/ofxPS3EyeGrabber/blob/master/src/ofxPS3EyeGrabber.cpp#L372-L391

[charlesveasey]

yeah i'm not sure how you can read the native pixel format from the track, but I know you can read the codec.

I was doing that sometime ago and falling back on QuickTime when not H264 or ProRes: https://github.com/charlesveasey/ofxAVFVideoPlayer/commit/df8895922b34c70cd9fe9d356d41454288779ea4

Generally it seems:

iOS: h264: kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange iOS: JPEG: kCVPixelFormatType_420YpCbCr8BiPlanarFullRange

OS X: Pro Res: kCVPixelFormatType_422YpCbCr8 OS X: h264: kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange

To get ProRes type specific then I think it is bit more difficult as mentioned here: https://developer.apple.com/library/mac/documentation/AVFoundation/Reference/AVAssetReaderTrackOutput_Class/index.html

http://stackoverflow.com/questions/13950076/how-to-get-supported-pixel-format-of-video-capture-device-camera-in-mac-os-x

[charlesveasey]

When looking at codecs I get codes like: ("apch"); // pro res ("avc1"); // h264

Though the last link from stackoverflow suggests that you can get the native Pixel Format by calling CMIOObjectGetPropertyData

And the corresponding CMPixelFormatType enum is available from Apple.

https://developer.apple.com/library/mac/documentation/CoreMedia/Reference/CMFormatDescription/#//apple_ref/c/tdef/CMPixelFormatType

[i-n-g-o]

thanks for the input and the links.

changed it back to pixelFormat selection based on the codec information for OF_PIXELS_NATIVE and updated some pixelformats.

i encounter problems using apples recommendations (https://developer.apple.com/library/mac/documentation/AVFoundation/Reference/AVAssetReaderTrackOutput_Class/index.html):

OS X: h264: kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange does not work: CVOpenGLTextureCacheCreateTextureFromImage() fails with error -6683.

OS X: ProRes 4444 with alpha (kCMFormatDescriptionExtension_Depth has value 32) with either kCVPixelFormatType_4444YpCbCrA8, kCVPixelFormatType_4444AYpCbCr16 or kCVPixelFormatType_64ARGB fails the same error -6683.

supporting yuv with alpha would be cool though. any ideas?

[arturoc]

in case it's helpful for testing, when i implemented this for gstreamer the most common native format for most compressed video formats is I420

[i-n-g-o]

i think it comes down to using kCVPixelFormatType_422YpCbCr8 for non-alpha movies, and kCVPixelFormatType_32RGBA (kCVPixelFormatType_32BGRA respectilvely for ios) for movies with alpha.

the mapping of ofPixelFormat to the internal format can respect I420 (kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange). it just might be, that AVFoundation cannot then uncompress the frames.

[arturoc]

we can also add more pixel formats and shaders if it's needed. the idea is that the file itself is ina certain colorspace before encoding it so using that colorspace is the most optimal as you avoid doing any colorspace conversion (in the cpu). for movies with alpha you could even use yuva which is a 4:2:0 + alpha

[dimitre]

There were some recent changes in AVFoundationVideoPlayer and for my usage it improved. It would be great to have your test and opinion to see if this issue can be closed.