search

gianni-rosato / svt-av1-psy

The Scalable Video Technology for AV1 (SVT-AV1 Encoder and Decoder) with perceptual enhancements for psychovisually optimal AV1 encoding
https://svt-av1-psy.com
BSD 3-Clause Clear License
243 stars 19 forks source link

[BUG] Bug on keyframes with tf enabled #55

Closed ItachiUchiha-IU closed 1 month ago

ItachiUchiha-IU commented 5 months ago

I just made an issue report on the gitlab of the mainline version of svt-av1, as this issue is present in all versions of both mainline and psy (SVT-AV1-PSY v2.0.0-9-gc76ddcfd, SVT-AV1-PSY v2.1.0 etc.). I'm reporting this to you guys too as you may be more interested in this, as you're way more attentive to quality than the mainline. But if this issue is something that the mainline should fix, then take this just as information. Here's the link to the issue report: Bug on keyframes with tf enabled

The issue is as stated in the title: with tf enabled, the keyframes are bugged. In most cases the picture is completely washed of all the details, and this continues in the following 4~12 frames, then it reached the best quality that remains until the next keyframe. In other cases there are huge artifacts, and this gets completely fixed in the exact following frame.

I made some comparison to make it easy to see: https://slow.pics/c/3gUTULwA

As you can see in the 4th image, when enabling tf, there's a gigantic artifact on the right (the place is completely random, I've found one right in the middle of the frame), which, as said, gets fixed completely in the following frame.

In pictures 1,2,3 you can see the 'details washing', but in this case it takes 4, 8 or 12 frames to fix the image (I'm not sure if it's comprehensible what i just wrote, so I'll rewrite it as a spoken commentary from what I can see looking at the video frame by frame: looking at the keyframe, it's easily noticeable how the keyframe lacks in details, but then, at the 4th or 8th or 12th frame after the keyframe the image suddently becomes way more detailed, and then it retains that level of details until a new keyframe comes).

The clips are downloadable from the gitlab post, otherwise here I repost the links that download the files uploaded on gitlab: direct download from gitlab - orig_cut1_Shigatsu_e01.mkv direct download from gitlab - orig_cut2_Shigatsu_e01.mkv

Notes: I tried changing several multiple settings (tune, mode, overlays, restoration, cdef etc) but nothing seems to fix this issue.

I conclude saying my huge thanks for all your work, as the psy version is rly something admirable.

gianni-rosato commented 5 months ago

Hi,

Thanks so much for your detailed issue report! For now, we're likely going to wait until we see a response from mainline SVT-AV1 on this & take some kind of action depending on how they proceed. In the meantime, it may be helpful to provide the exact settings you used & with which tool (FFmpeg, SvtAv1EncApp, etc) to both this issue & mainline's issue report to make troubleshooting easier. I hope this gets resolved, for both of our sakes!

ItachiUchiha-IU commented 5 months ago

Thank you for the fast reply! I absolutely agree, it rly should be something the mainline guys fix themselves, and I rly hope they do, avoiding you guys any trouble/work (in case you decide to look into it yourselves if they don't).

The issue appears to be caused directly by temporal filtering, so it should happen with any kind of settings combination or tool. In short, from my perspective, whenever tf is enabled, the issue occurs, no matter the settings/tools.

I tested (on Windows) with crf 7, 11 and 12 with the following tools and settings:

I updated the manline's issue report too. Thank you again!

gianni-rosato commented 1 month ago

I think this issue is out of scope for our project, so for now I'll tentatively close this. If you're experiencing the same issue, please see the mainline bug report

ItachiUchiha-IU commented 1 month ago

I think the same, as I realized that the whole tf matter is way bigger than what it seemed at first: it would require a complete or at least "partial" (probably very big part) rework of it (if Im not wrong interpreting how it works).

Btw I was testing the new tf-strength setting just in these days (as I sadly couldnt before) and I can say that being able to adjust the strength of the tf is already a good solution to mitigate the problem on keyframes and the few following frames sometimes affected. Furthermore, it allows even more tuning, thus allowing even more control on the final result of the encode and not resulting in the user having to completely disable the tf (which was the worse option, as it made the file substantially bigger in size and worse in visual quality often without even normalizing the bitrate, at least at the lower crf ranges that I use and test).

So yeah, with the addition of tf-strength I would say anyway that the specific bug on the tf can now be considered unimportant (especially for the psy project) as tf-strength substantially did the trick in improving the tuning and usage of the tf !

Personally I think it could be useful to mainline too (at least as partial workaround to the real problem), so idk if u guys want to propose it in the mainline bug report or as merge request or somewhere else or if u're simply not interested. In case u want to do it, I just advise to propose it with strength 3 as default instead of 1 as I was said that's the strength that tf had before the addition of the strength, thus getting the same file as before if Im correct.

As always, thanks for all your work, I rly appreciate it.

silverbacknet commented 1 month ago

Another potential related addition might be a separate tf-strength-keyframes setting, or simply a ratio to turn tf way down on keyframes.