videos from `RGB{N0f8}` matrices are unreadable by most players

[ExpandingMan]

Creating a video with e.g.

map(_ -> rand(RGB{N0f8}, 100, 100), 1:60)

results in a file which is unreadable by most video players. For example, VLC gives

[h264 @ 0x7fa498cad2c0] hardware accelerator failed to decode picture

This is the case for every format I've tried including mkv and mp4.

[galenlynch]

Could you please post more information about your version and how you are encoding?

My rgb videos work fine, even with hardware acceleration.

[ExpandingMan]

julia> using VideoIO, Images;

julia> v = map(_ -> rand(RGB{N0f8}, 128, 128), 1:60);

julia> VideoIO.save("testfile.mkv", v);

julia> VideoIO.save("testfile.mp4", v);

julia> versioninfo()
Julia Version 1.6.1
Commit 6aaedecc44 (2021-04-23 05:59 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-5930K CPU @ 3.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, haswell)
Environment:
  JULIA_BINDIR = /opt/julia/bin
  JULIA_NUM_THREADS = 6
  JULIA_EDITOR = nvim

Both of these videos seem to be corrupt according to VLC, and many other programs (in particular all the native android video players) don't seem to be able to read them either. This is on VideoIO 0.9.2.

[galenlynch]

Weird, I just tried your example (mp4 at least) and it's fine and plays with HW acceleration in VLC, gnome videos, mpv, etc...

Could you post an example of your corrupted videos?

[galenlynch]

Oh you know what, I just noticed that the default video format for rgb videos is yuv444, which might not be supported on all players or by most hardware acceleration. Does it work if you turn off hardware acceleration?

[ExpandingMan]

I won't get a chance to try it again until a little bit later, but I just wanted to comment that it's definitely not just VLC that can't view the videos. I could not see them on the default video player on an android device, and I have some friends who tried to view them on phones and were told they were corrupted (I'm not sure what programs they used). I'll check in a little bit if I can watch them on VLC without hardware acceleration. mpv seems able to view just about anything.

[galenlynch]

You might need to encode them with yuv420 for some video players, which will incur chroma downsampling. Currently VideoIO just allows FFMPEG to select the "best" pixel format (in terms of minimizing quality loss), which might not be the most broadly compatible. If MPV plays it, then it's likely not corrupted, just not a format that these players will accept. You can set the target pixel format using the target_pix_fmt keyword option, e.g. target_pix_fmt = VideoIO.AV_PIX_FMT_YUV420P.

[ExpandingMan]

Ok thanks, I'll experiment with that some.

Would you be open to changing default settings to something broadly compatible? Seems like something most users will want. Given the prominence of VLC it might be a good source of guidance, though we should check others as well.

[galenlynch]

Yeah I can repeat your problem when I turn on HW acceleration for VLC, it doesn't seem to fall back to software decoding. If I set the pixel format to yuv420 then it works on my pixel 2 and VLC with hardware decode.

NVidia doesn't currently support h264 YUV444 decode AFAIK (though it does with HEVC). Not sure about other hw decode accelerators (e.g. intel, whatever your adroid phone has). VLC however does play it with software decode.

Though I was pretty involved in revamping the encoding/decoding, I'm not actually a mantainer here so can't speak to which defaults should be used. 420 inherently degrades the video due to chroma downsampling, but has good support. Unfortunately the combination of encoder, pixel format, playback software, and hw acceleration makes it very hard to pick a good default for all cases. I think this package currently tries to walk a fine line between being a simple interface to FFMPEG (e.g. letting it pick the pixel format here instead of VideoIO deciding), and an opinionated package that "works for most people." Maybe in this case we could hard-code a default that sacrifices image quality for android / hw acceleration compatibility, but I'll leave that to @IanButterworth.

After tooling around in the guts of this package for a while, I think it might make sense (very long term) to split this into three packages: a package that's just wrapping the FFMPEG library (i.e. just auto generated functions for each library function), a package that is unopinionated and handles some of the boring bits of encoding like setting up buffers etc, and a package that is very high level for people that have some array and just want to make a movie and don't know or care about pixel formats and hardware decode capabilities etc.

[galenlynch]

Maybe this is a documentation issue as well, perhaps we could put a section on which encoding settings to use if you want a video to play most places and are ok with quality loss.

[ExpandingMan]

Ok, got a chance to mess around with this.

The pixel format definitely seems like it was the principle issue. As far as I can tell everything can read with AV_PIX_FMT_YUV240P, though how happy a given program seems to be about it would seem to depend on various settings including the container type.

We certainly are in dire need of documentation at the very least. I like to think I'm pretty knowledgeable about computing and general programming, but I've never been near videos before and the plethora of options available from ffmpeg go WAY over my head, so making intelligent choices here is extremely difficult for me.

Surely if the default for VideoIO as it is right now is lossless compression, the default will have to stay that way, however I'm sure many people would find it useful if there were some sort of compatsave (or something) that had default settings for outputting videos that just about everything is likely to be able to read.

[lawless-m]

Can I just add that YUV444 is also not playable in Firefox

https://bugzilla.mozilla.org/show_bug.cgi?id=1368063

It took me 3 hours to find this post containing the solution - which was when I came to report you couldn't do the equivalent of

ffmpeg.exe -i img_%06d.png -pix_fmt yuv420p img.mp4

I was looking too hard at Encoder Options,

So glad I found it though. Thanks for all of your work.

[IanButterworth]

Does anyone here think they could do a first pass at the new documentation section proposed?