Wrong colors / YUV > RGB conversion

[abolibibelot1980]

Hi,

I want to generate thumbnails previews for a very large number or videos, before sorting them out. I tested VCSI and a few other tools for that purpose, but I noticed that most of them got the colors wrong, including VCSI, MTN, Scorp Video Thumbnails Maker, MPC-HC ; only SMPlayer got the colors right, but it doesn't allow to generate those previews in batch, by treating an entire directory, or to be launched by command line, so I can't practically consider using it for this task. The source files are 1920x1080 AVI / GAVC, and the color conversion matrix should be Bt.709, but apparently ffmpeg uses the Bt.601 by default. I asked for guidance on that matter on VideoHelp forum, ran some tests doing screenshots with ffmpeg and the suggested switches : I obtained the right colors (from a video I edited showing at some point a still picture which I took of a red car, so I could easily see if the colors were right or wrong) with the following switch : -vf zscale=matrixin=709:matrix=709,format=rgb24 Now the question is : how could I modify the current VCSI script to integrate that switch so as to obtain thumbnail previews with accurate colors ? I don't know anything about the Python language, but looking at the script I can see that there are three instances of ffmpeg calls, apparently which one is actually used depends upon certain parameters of the frame, which I don't quite understand at that point. Will it work if I just add that switch to those three ffmpeg calls and install the modified script with a new name ?

Side question : I also noticed that the thumbnails generated by MPC-HC were slightly sharper than the others (and those generated by MTN definitely blurrier – perhaps it creates JPEG-compressed temporary files, I can't say, it comes as an already compiled .exe and hasn't been updated since 2008 so I doubt that I could reach the author...). Perhaps this could be improved in VCSI by adding a ffmpeg switch to resize with the Lanczos algorithm instead of the default one ?

Another small quibble : the fields available for the file information header are less complete than with MTN : with the video I used for those tests, MTN can display the general average bitrate, the video average bitrate, and the precise type of codec / format used, namely “h264 (High) (GAVC / 0x43564147), yuvj420p (pc, progressive)“, whereas with the current VCSI script no video bitrate field is proposed in the options, and the format is more broadly designated as “H.264 / AVC / MPEG-4 AVC”. VCSI is still better than MPC-HC and SMPlayer in that regard. The Scorp tool is in between in that regard, displaying “GAVC” (which seems more specific than “AVC”) and the video average bitrate, but with the free licence it puts a giant logo on each generated image, and the colors are also wrong, so it's not usable. Right now VCSI looks like the best option.

By the way, what does the “I” stand for in “VCSI” ?

It would be nice to have a quick feedback, at least regarding the main issue (color shifting), so I could get started with this task which is going to be a long hassle in and of itself ! :^p Thanks in advance.

The relevant threads on VideoHelp : https://forum.videohelp.com/threads/390691-Generating-video-thumbnails-preview-contact-sheet-in-batch-color-issue https://forum.videohelp.com/threads/390710-Screenshots-and-respect-of-the-YUV-RGB-colorspace-conversion

[abolibibelot1980]

Regarding the main question : I managed to get correct colors by adding this line to all three instances of ffmpeg calls in the vcsi.py script, then re-installing it : "-vf", "zscale=matrixin=709:matrix=709,format=rgb24", (At first I wanted to install the modified script side-by-side with the native one, by giving it another name in setup.py and some other files, but it didn't work, I got a “ModuleNotFoundError”.) So it looks like this : ffmpeg_command = [ "ffmpeg", "-ss", skip_time, "-i", self.path, "-ss", skip_delay, "-vframes", "1", "-vf", "zscale=matrixin=709:matrix=709,format=rgb24", "-s", "%sx%s" % (width, height), ]

And here's the result : => made with the current vcsi script => made with the modified script, using Bt.709 matrix

You can see that the greens are less “flashy” and the reds more vivid, same as the preview generated by SMPlayer. So this should be the default behaviour for “HD” footage (starting from 1280 pixels in width), and there should be an added option to select another color conversion matrix if needed (for instance Canon DSLR cameras are known to generate footage with the Bt.601 matrix).

I can run the intended task with this modified script, but I would still appreciate a feedback regarding this and the other issues I mentioned.

[Revan654]

Hi,

I want to generate thumbnails previews for a very large number or videos, before sorting them out. I tested VCSI and a few other tools for that purpose, but I noticed that most of them got the colors wrong, including VCSI, MTN, Scorp Video Thumbnails Maker, MPC-HC ; only SMPlayer got the colors right, but it doesn't allow to generate those previews in batch, by treating an entire directory, or to be launched by command line, so I can't practically consider using it for this task. The source files are 1920x1080 AVI / GAVC, and the color conversion matrix should be Bt.709, but apparently ffmpeg uses the Bt.601 by default.

Many videos don't have Color flags in there metadata, FFMPEG only pass the metadata through, it doesn't convert the Matrix, transfer or prime colors.

-vf zscale=matrixin=709:matrix=709,format=rgb24 Now the question is : how could I modify the current VCSI script to integrate that switch so as to obtain thumbnail previews with accurate colors ? I don't know anything about the Python language, but looking at the script I can see that there are three instances of ffmpeg calls, apparently which one is actually used depends upon certain parameters of the frame, which I don't quite understand at that point. Will it work if I just add that switch to those three ffmpeg calls and install the modified script with a new name ?

Yes, As long as it has proper FFMPEG format(Filters need to be in quotes).

Example: "-vf", "fps=15,scale=480:-1:flags=spline" if you want cmdline arg control you need to add code to the opts.

Link: https://github.com/amietn/vcsi/blob/eb436cc2aee268c66f7fa36e663ccf6a772c4a3b/vcsi/vcsi.py#L364

Side question : I also noticed that the thumbnails generated by MPC-HC were slightly sharper than the others (and those generated by MTN definitely blurrier – perhaps it creates JPEG-compressed temporary files, I can't say, it comes as an already compiled .exe and hasn't been updated since 2008 so I doubt that I could reach the author...). Perhaps this could be improved in VCSI by adding a ffmpeg switch to resize with the Lanczos algorithm instead of the default one ?

bicubic is ffmpegs default resizer, Lanczos is very similar to Bicubic. Spline would be a better choice since it contains better AA. The jpeg compression will always reduce quality, PNG is better format to use, filesize will be slightly larger.

Another small quibble : the fields available for the file information header are less complete than with MTN : with the video I used for those tests, MTN can display the general average bitrate, the video average bitrate, and the precise type of codec / format used, namely “h264 (High) (GAVC / 0x43564147), yuvj420p (pc, progressive)“,

Not sure on the reason why it lacks these fields since FFprobe does have some of them. Which can be easily added.

MTN Creates some of it's headers from scratch and doesn't count on external resources.

You can also use Mediainfo & MediaInfo DevKit that has allot more details on the video. Then Build your own.

If you want MTN with Newer Codec Support: https://github.com/Revan654/movie-thumbnailer-mtn It just needs Compiled, I haven't gotten to around to compile it yet for Release.

[abolibibelot1980]

Thanks for these detailed replies.

Many videos don't have Color flags in there metadata, FFMPEG only pass the metadata through, it doesn't convert the Matrix, transfer or prime colors.

But isn't it a widespread convention that "HD" footage should be treated with the Bt.709 matrix ?

bicubic is ffmpegs default resizer, Lanczos is very similar to Bicubic. Spline would be a better choice since it contains better AA. The jpeg compression will always reduce quality, PNG is better format to use, filesize will be slightly larger.

What does "AA" stand for here ? I compared the outputs in JPEG (which is the only format available in MTN), at the same quality setting (90), and clearly MPC-HC produces the sharpest, most vivid and generally most pleasant thumbnails (although the aspect ratio is slightly off and offers no option to fine-tune this), while MTN's output (with the 4.0 version included in StaxRip -- the 200808a version doesn't work with the test footage) is definitely blurry, even at JPEG maximum quality (-j 100). I wonder if it creates JPEG-compressed temporary files, as suggested above, or if this is caused by an outdated decoding or encoding engine... I found MTN's source code on Github since I wrote this, but I can't find the command which generates the tiles so I don't have a clue. One thing about MPC-HC's output is that it's influenced by the settings of the graphic driver, which is not the case for the other tools (I deactivated some “enhancements” in Intel graphic settings and it immediately changed the picture aspect as well as the screenshots). Is it because it is relying on DirectShow ?

=> MPC-HC

=> MTN (JPEG q90)

Yet MTN is also by far the fastest, among those who can treat a whole directory in batch. As a test, I ran them on a folder containing about 350 videos, AVI format, ~112MB each, and stopped after 60 seconds, here is the number of preview pictures generated by each tool during that time : VCSI => 17 Scorp Video Thumbnails Maker => 40 with the default “crystal” engine, and 36 with the “ultimate” engine (although it's supposed to be faster) MTN => 104

So, ideally, I'd like to get the speed of MTN with at least the quality of VCSI, and ideally the quality of MPC-HC, and an option for selecting the color conversion method.

You can also use Mediainfo & MediaInfo DevKit that has allot more details on the video. Then Build your own.

The info part is a nice refinement but not a priority. The options provided by MTN are enough for my purposes. If detailed informations are needed it's better to create TXT files with MediaInfo CLI.

If you want MTN with Newer Codec Support: https://github.com/Revan654/movie-thumbnailer-mtn It just needs Compiled, I haven't gotten to around to compile it yet for Release.

How should I proceed to compile it ? Is it likely to improve the output quality, compared with the version I've tried so far ?

[abolibibelot1980]

So I tried this modified MTN, and to my surprise the files it generates are exactly the same (same size / checksum) at the same settings as those generated by the older version relying on older libraries... If you worked on the actual code, how does it proceed to create the tiles / thumbnails, and could this behaviour be improved somehow ? Perhaps it's not the best place to discuss about another program, and I should open a new issue for MTN instead... But it's frustrating since each of the 5 tools I've tried for this task does something right, yet none of them meets all the important criterions.

[Revan654]

But isn't it a widespread convention that "HD" footage should be treated with the Bt.709 matrix ?

Should, Yes. However I seen countless HD/4K videos use BT.601 / bt470m instead.

What does "AA" stand for here ? Anti-Aliasing, It cleans up the edges in the picture.

I compared the outputs in JPEG (which is the only format available in MTN)

^From Internal Thumbnail Creator

I assume you mean the old MTN, The version that is included with StaxRip has support for PNG.

Default format is jpg since it's faster to create then PNG.

MPC-HC

It's likely adding an Sharpener, along with AA among a few other filters to make it look cleaner & Colorful. Plus converting the color to RGB before capture.

included in StaxRip -- the 200808a version doesn't work with the test footage) is definitely blurry, even at JPEG maximum quality (-j 100).

jpg adds compression, You want a lossless format like PNG or Tiff.

Try the internal StaxRip Thumbnail Maker(Version 2.0). It should be allot cleaner, Since the captures are not passed a Encoder that can reduce the quality of the capture, but created from the ground up internally.

MPC & DirectShow

It's very likely MPC is DirectShow Driven Application. You Might want to Check out PotPlayer which also has Thumbnailer Creator.

You can also make one with MPV too with a small script.

Yet MTN is also by far the fastest, among those who can treat a whole directory in batch.

MTN is written in C# and only uses the dll files when needed, C# is the fastest code around. Python and such has to compile first before start then has to wait for FFMPEG.exe to capture each from. Which is longer process. It also depends on FFMPEG build too and how it was compiled.

So, ideally, I'd like to get the speed of MTN with at least the quality of VCSI, and ideally the quality of MPC-HC, and an option for selecting the color conversion method.

Try StaxRip internal Thumbnail Maker, It uses C type Coding & has detail header. It also has support for Lossless formats and uncompressed format. The Next Build will include tweaks for colorspace & such.

It does Require Avisynth FrameServer to be installed.

How should I proceed to compile it ? Is it likely to improve the output quality, compared with the version I've tried so far ?

That Build was put together very quickly. I've cut down External Files needed, Only FFMPEG & GD files are now needed.

For Window Builds you need MinGW, FFMPEG Dev & Shared Files(32 Bits) + LibGD.

I suggest Making a Folder Directly on the C Drive and Name it something like MinGW.

Once you have all the files in the correct location & Path are correct you just need to type make -f Makefile.mingw The software will do the rest.

---

I forgot I even did this, This is a tweaked version with MediaInfo Support, Removed a few lines and This might have the correct FrameRate data(Certain FrameRates will return Null results).

I also removed some of External modules in favor for pre-installed modules which might have increased the speed a bit.

Changes a few of the defaults values.

I also removed allot of the text that prints to screen, Which got annoying.

This is the Result Now:

[abolibibelot1980]

I assume you mean the old MTN, The version that is included with StaxRip has support for PNG.

Yes, I've discovered this when doing further testing with your fork. The integrated help or the descriptions on GitHub don't explicitly mention that possibility.

Try the internal StaxRip Thumbnail Maker(Version 2.0). It should be allot cleaner, Since the captures are not passed a Encoder that can reduce the quality of the capture, but created from the ground up internally.

Already tried (StaxRip 1.9.0.0 which is the latest stable release), it doesn't work with the kind of file format I have to inventory (AVC in AVI container), I get this error :

InvalidCastException (1.9.0.0) La conversion de la chaîne "" en type 'Double' n'est pas valide.
System.InvalidCastException: La conversion de la chaîne "" en type 'Double' n'est pas valide. 
---> System.FormatException: Le format de la chaîne d'entrée est incorrect.

With a random MKV file (HEVC / 5.1 AAC) it works but makes StaxRip crash right afterward. Works fine with a small MP4 file. Also this tool doesn't seem to allow batch-processing, which is mandatory as there are thousands of videos to process.

jpg adds compression, You want a lossless format like PNG or Tiff.

Again, compression is not the issue here. JPEG output from VCSI or the other tools looks good enough, while even PNG output from MTN lacks detail in comparison. It would probably go unnoticed for small thumbnails like the examples you provided, but with 4 tiles of 960x540 pixels it's quite conspicuous.

You can also make one with MPV too with a small script.

Could you please elaborate ?

I forgot I even did this, This is a tweaked version with MediaInfo Support, Removed a few lines and This might have the correct FrameRate data(Certain FrameRates will return Null results).

Well, where is this other tweaked version ? Or you mean the one you linked earlier ? (This one I already tried, see my previous post, or the issue I opened on the dedicated page.)

I also removed allot of the text that prints to screen, Which got annoying.

Speaking of which, running MTN with those AVI/AVC files I got many warnings like “data partitioning is not implemented, update your FFmpeg version...” or “deprecated pixel format used, make sure you did set range correctly” ; it works anyway but is annoying indeed.

[Revan654]

Already tried (StaxRip 1.9.0.0 which is the latest stable release), it doesn't work with the kind of file format I have to inventory (AVC in AVI container), I get this error :

Staxrip accepts all forms of AVC.

With a random MKV file (HEVC / 5.1 AAC) it works but makes StaxRip crash right afterward. Works fine with a small MP4 file. Also this tool doesn't seem to allow batch-processing, which is mandatory as there are thousands of videos to process.

Like I said Try 2.0, Since there was some small issues with Length Names & Dotnet not wanting to accept them.

Speaking of which, running MTN with those AVI/AVC files I got many warnings like “data partitioning is not implemented, update your FFmpeg version...” or “deprecated pixel format used, make sure you did set range correctly” ; it works anyway but is annoying indeed.

That's beyond my control, That all at the end of FFMPEG dll and How FFMPEG reads everything.

Well, where is this other tweaked version ? Or you mean the one you linked earlier ? (This one I already tried, see my previous post, or the issue I opened on the dedicated page.)

I'm Talking about a Tweaked version of VCSi.

With a random MKV file (HEVC / 5.1 AAC) it works but makes StaxRip crash right afterward. Works fine with a small MP4 file. Also this tool doesn't seem to allow batch-processing, which is mandatory as there are thousands of videos to process.

It Uses MultiSelect, You can Select all the files you want.

Again, compression is not the issue here. JPEG output from VCSI or the other tools looks good enough, while even PNG output from MTN lacks detail in comparison. It would probably go unnoticed for small thumbnails like the examples you provided, but with 4 tiles of 960x540 pixels it's quite conspicuous.

If you want larger tiles adjust the optargs. 2x2 is useless to have since it only grabs 4 shots and doesn't show the entire clip.

[abolibibelot1980]

Like I said Try 2.0, Since there was some small issues with Length Names & Dotnet not wanting to accept them.

Sorry, I didn't realize that you were one of the main authors of StaxRip, so you definitely know where it's at ! :) And I downloaded version 1.9.0.0 on October 26th, it happens that the new version was released just that day...

It Uses MultiSelect, You can Select all the files you want.

What about recursive directory processing ? I'll try anyway...

I'm Talking about a Tweaked version of VCSi.

Oh, sorry for the confusion... but where can I find this tweaked VCSi ?

If you want larger tiles adjust the optargs. 2x2 is useless to have since it only grabs 4 shots and doesn't show the entire clip.

Those are short videos, less than 2min. so it's enough for that particular purpose.

[Revan654]

What about recursive directory processing ? I'll try anyway...

I'll get around to one Day, New Functions have to be built for Opening a Folder Browser with Thumbnail Creator, Then Function for Searching inside Given Folder. It's allot of Code that has to be added, Plus I have other Projects I'm working on.

Oh, sorry for the confusion... but where can I find this tweaked VCSi ?

You need all three py Files + MediaInfo DLL File.

https://gist.github.com/Revan654/bb36f17593d4a7d3e046967dc46f01ef

[amietn]

@abolibibelot1980 Sorry for such a late reply.

Thank you for opening this issue.

If I understand correctly, you would like:

• A possibility to pass arbitrary parameters to the ffmpeg command used internally. For example something similar to vcsi --extra-ffmpeg-args '-vf zscale=matrixin=709:matrix=709,format=rgb24' foobar.mp4
• More exposed metadata fields, such as the video bitrate.

Is that correct?

Some clarifications:

• The i in vcsi does not have a specific meaning.
• Video metadata comes directly from the output of ffprobe as seen in this sample output. Any information in that output could be easily exposed.

I hope this answers some of your questions.

[remlap]

I would love an extra ffmpeg arguments flag to with the increasing HDR content.