AVFoundation player doesn't load HDV ( non-square pixels ) correctly.

[ofTheo]

https://forum.openframeworks.cc/t/issues-with-pixel-ratios-in-ofvideoplayer/22120/10

Test video is here: https://www.dropbox.com/sh/t2rezaeeghyj581/AAAAkGZvI6uJmG6KYOKEFCNUa?dl=0

On initial research the way to get the true size of the frame is:

CVImageBufferRef imageBuffer = [videoPlayer getCurrentFrame];
  int realW =  CVPixelBufferGetWidth(imageBuffer);
  int realH = CVPixelBufferGetHeight(imageBuffer);

The problem is that this info comes in later ( once the first frame is delivered ). So for texture playback we need to change the texture coords based on the difference between the reported width ( 1920 ) and the pixel width ( 1440 )

For getting the pixels out we would need to figure out whether we should return the pixels as they are ( 1440 x 1080 ), or scale the pixels up which would add overhead.

ping @i-n-g-o @openframeworks/macos

[ofTheo]

So this commit partially fixes the issue. The video player now knows that its width is 1440 and not 1920 - so the texture doesn't draw unallocated pixels and getPixels() doesn't crash anymore because the image buffer and getWidth() / getHeight() are now the same. https://github.com/openframeworks/openFrameworks/commit/4834bcd4db91c681ea8fbb7d02f1055b3b61b66c

However the video is drawn squished at 1440 x 1080. I added in two params for displayed width and height which in this case would be 1920 x 1080.

The issue though is should getWidth() be the the size of the pixels and the texture or the size it is meant to be drawn at? It might be strange if video.getWidth() is 1920 but video.getPixels().getWidth() is 1440;

If video.getWidth() and video.getPixels().getWidth() are 1440 and we have video.getDisplayWidth() be 1920 then it could be easier to accidentally lay things out wrong if you use video.getWidth();

Anyway sort of thinking out load here. :)

If we're fine with a video that is displayed 1920x1080 but consists of 1440x1080 pixels showing up as a 1440x1080 video and the user having to draw it with the correct ratio ( which we can provide them ), then the commit above, plus adding access to the display W / H will close this issue.

[fred-dev]

It could be substantial work in adding a clean way to deal with this, anything pixel based needs to be adjusted for the non square pixels- if the draw is fixed that is great, if the pixels are fed into openCV it will return features that are based on square pixels and if a contour finder is drawn over the top of a movie player that is adjusted for the aspect ratio the coordinates will not match. I guess as openCV is part of the core this is an issue. (I will test anamorphic live capture of 16:9 PAL images with ofVideoGrabber to check for the same error).

I have posted on a video post forum out of curiosity to check about the future of non-square pixels just to see if it is a thing from the past or if it is something to deal with in the future.

video.getDisplayWidth() seems logical, I could also imagine (if it seems that these will become legacy formats in the near future) it could be also video.getIsAnamorphic(). This would leave a lot of work to the users but the handling is simple after that. In this case the real pixel widths and heights could be returned and then other pixel based operations will return coordinates consistent with the draw. It is then up to the user to adjust the aspect ratio for display purposes.

Out of curiosity, can openCV find a chessboard on non-square pixel based images?

[ofTheo]

Thanks @fred-dev for now I made the changes just to the ofAVFoundationPlayer class - but if this is something that might be needed on other platforms we could add it to the ofVideoPlayer class.

Take a look at #4949

Thanks for your help!

[hamoid]

I'm having a similar issue on Linux. Should I open a new issue?

ffprobe /home/me/Videos/2016/MAH05408.MP4
Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1440x1080 [SAR 4:3 DAR 16:9], 11997 kb/s, 25 fps, 25 tbr, 25k tbn, 50 tbc (default)

Note the 4:3 and 16:9.

In VLC the video plays right, but says display resolution is 1440x1080. If I take a screenshot and calculate the actual size in gimp, I see it's actually playing at 1440x810!

In OF, it seems to be playing by 1440x1080, and therefore it looks stretched vertically.

[keikoro]

New to OF and seem to have a same/similar/related problem as the original one (arrived here via this thread), also on Mac.

The mp4 I'm trying to play is drawn the same size as it is played in VLC (videoFile.getWidth() and videoFile.getHeight() correspond to the size of a screenshot taken of the video playing in VLC). But, the video is squashed to two thirds of it width to the left; the rightmost third is just lines of pixels (the last pixels on the right repeating).

So... has this issue been fixed in theory but the PR gone unmerged for over 2 years? Is there another fix? What are the next steps here?

(Tbqh, the fact I've run into several issues already within the first couple of hours of using – or trying to use – OF, including buggy examples, broken links in the docs, this issue,... isn't particularly motivating.)

[arturoc]

@kerstin can you try if the nightly builds fix your issue? we are about to release a new version but this bug was fixed after the last released happened so 0.9.8 still has it