Apply FFmpeg filters on videos

[veikk0]

Here's my problem/use case.

I have a file I've already transcoded with FFmpeg, and in the process of doing that I applied deinterlacing. When calculating VMAF for this file I'd need to apply the same filter to the original file. As a workaround I could create a lossless version of the original file with the filter applied, but that'd take time and a lot of storage space.

Being able to apply FFmpeg filters would also be useful for the other comparison modes, since there are many reasons one would apply FFmpeg filters to improve a video.

[CrypticSignal]

What is the purpose? To see the effect of certain filters on VMAF, or just to have the ability of adding certain filters with the purpose still being to compare presets/CRF values (in which case a lossless version of the original video will need to be created)? If it's the former, I'm not sure if this is a good idea. I don't know much about FFmpeg video filters as it's not something I've delved into, but don't they change the video in a substantial way, making the absolute VMAF scores pointless? If it's the latter, I don't see the point either. You'd still be comparing the same thing (presets or CRF values), but you're most likely increasing the completion time as a filter is being added to the command.

I suppose I'm not totally opposed to this idea, so perhaps you can elaborate on why you think this would be a good addition to VQM. This (video filters) is not my area of expertise.

I have a file I've already transcoded with FFmpeg, and in the process of doing that I applied deinterlacing.

I'm assuming you used -vf yadif?

[veikk0]

I'm assuming you used -vf yadif?

I used bwdif, which actually doubles the framerate too (by default it outputs one frame per field).

The point is being able to compare files at all by removing the effect of filters from the equation. To measure compression efficiency, the reference material needs to be exactly the same as what was provided to the encoder that created the file for which VMAF is being calculated.

As for why one would use filters. There is material that I would never encode as-is without applying filters, because I would never consume (or deliver the video to be consumed) with distracting and unnecessary artifacts such as interlacing. So why would I encode and calculate VMAF for an unfiltered file if I deemed the result unacceptable anyway? Such artifacts can also affect compression efficiency, which is another reason to apply filters to mitigate them.

Cropping would be a more widely used example. If I'm re-encoding a widescreen film from a letterboxed Blu-ray source and cropping out the black bars, I'll end up with video dimensions that differ from the source video (often something like 1920*800 vs the 1920*1080 of the original). This will make the source file useless for a VMAF comparison unless I can crop it, because the difference in video dimensions screws with the VMAF calculation. In fact, the FFmpeg implementation of the VMAF filter will exit with an error complaining that the video dimensions aren't the same.

Currently when using VQM, the workaround would be creating a lossless cropped version of the source file for comparison purposes (which is a PITA and also takes time). Not cropping isn't really a good option, because (in my experience) the uncropped area does negatively affect compression efficiency, at least with some encoders. I'd rather be able to pass the FFmpeg filters I used to VQM and have them applied for every score that needs to be calculated, despite any CPU cost involved, because IMO that's more convenient than having to deal with lossless intermediate files.

As an example, here's how I calculate VMAF with FFmpeg when the source file is uncropped and the encoded file is already cropped:

ffmpeg -r 24 -i encoded.mp4 -r 24 -i source.m2ts -an -sn -filter_complex "[1:v]crop=1920:800:0:140,libvmaf" -f null -

[CrypticSignal]

I'd rather be able to pass the FFmpeg filters I used to VQM and have them applied for every score that needs to be calculated, despite any CPU cost involved, because IMO that's more convenient than having to deal with lossless intermediate files.

Won't there still be a need to create a lossless version of the original with the filter applied, so that it can be used as the reference? Or does supplying the filter to -filter_complex mean that the filter is applied before VMAF is calculated, removing the need for a lossless intermediate file to be created?

Here's a simplified version of one of the commands that VQM will run:

ffmpeg -r 24 -i encoded.mkv -r 24 -i source.mkv \
-lavfi [0:v]setpts=PTS-STARTPTS[encoded]; \
       [1:v]setpts=PTS-STARTPTS[source]; \
       [encoded][source]libvmaf=n_subsample=1:n_threads=4 \
-f null -

Looking at the above command, do you know where the filter(s) would need to be added? I'm assuming like the following:

ffmpeg -r 24 -i encoded.mkv -r 24 -i source.mkv \
-lavfi [0:v]setpts=PTS-STARTPTS:crop=1920:800:0:140[encoded]; \
       [1:v]setpts=PTS-STARTPTS:crop=1920:800:0:140[source]; \
       [encoded][source]libvmaf=n_subsample=1:n_threads=4 \
-f null -

In case you don't know, -lavfi is the same thing as -filter_complex. You can use them interchangeably.

[veikk0]

does supplying the filter to -filter_complex mean that the filter is applied before VMAF is calculated, removing the need for a lossless intermediate file to be created?

Yes, that's what it does.

As for those examples. I've never gotten my head around filtergraphs and my example above is the only one I utilise on a regular basis (it's simple enough and I know that it works). One thing I do know about filters is that they're applied serially. So if the libvmaf filter is the last one in the chain, all other filters in the chain are applied before it.

The only fault I can see in that filtergraph is that the crop filter is being applied to both inputs. I'm assuming that encoded.mkv would already have been cropped at the encoding stage, so re-applying the fliter here wouldn't do anything.

[CrypticSignal]

Support for FFmpeg video filters has been added in commit https://github.com/BassThatHertz/video-quality-metrics/commit/6eb772b7f7186d5f9ae1b91e094ef8371959493b. Please pull the latest code, test it out, and report back. You need to use the -vf or --video-filter argument. Here's an example: python main.py -ovp original.mkv -crf 17 18 -p ultrafast -vf bwdif=mode=0,crop=1920:800:0:140

The only fault I can see in that filtergraph is that the crop filter is being applied to both inputs.

The other mistake that I had made is that I separated the setpts and crop filters with : instead of ,.

I'm assuming that encoded.mkv would already have been cropped at the encoding stage, so re-applying the fliter here wouldn't do anything.

I corrected that part. You can confirm by adding --show-commands when running VQM and looking underneath the Running the following command: message that is printed during the VMAF calculation stage and you will see that the filter is only applied to [1:v] which is the original video.

[veikk0]

Thank you very much for this feature. I just tried it and it works.

However, I did run into another bug: #35