Missing frames

[WeatherWonders]

I performed tests on the upscaling and interpolation separately using a short video after finding these issues with a longer video. I believe the input videos are not the cause after analysis with ffmpeg (extracting frames as PNGs) and ffprobe.

The upscaling output has two added black frames at the beginning, is missing the first frame of the input, and is missing the last two frames of the input. It has the same number of frames as the input.

The interpolation output (exactly twice the number of frames as the input) is as follows: First frame: black Second frame: interpolation between black and first input frame Last two frames: second-to-last input frame (no interpolation to, nor copy of, the last input frame)

It also seems that Enhancr applies lossy compression regardless of the output format, as the quality of subtle gradients in the video noticeably degraded, looking posterized and having jagged seams.

Enhancr version: 0.9.9 free Input video: exported from Blender sequencer as .mkv with AV1, 19 frames at 60 fps (ffprobe reports a duration of 0.38 seconds, 0.317 for the video stream specifically) Output format: Lossless frame sequence with no added ffmpeg parameters Upscaling method: RealESRGAN (DirectML) Interpolation method: RIFE 4.6 (NCNN)

[Bubblemint864]

Hello, post the mediainfo of your file (obtained from the app or https://mediainfo.js.org/). As for why lossless isn't actually lossless, it might be because the bit depth isn't set in the command - so maybe try adding -pix_fmt yuv420p10le if your input is 10 bit. Not sure why the black frames are appearing but might be explained by the mediainfo. Enhancr hasn't been updated a while so maybe the current version is unable to decode the frames correctly? Just a guess

Also just so you are aware, most Real-ESRGAN models are very destructive in terms of quality so proceed with caution.

[WeatherWonders]

Thank you very much for your help!

I had deleted the test file, so I made another one and tried again. It should be identical but this time only one frame was cut off from the end. The start is no different from the last time. Anyway, media info of that AV1 export:

General
    Format              Matroska
    Format_Version      4
    FileSize            9169545
    Duration            0.384
    OverallBitRate_Mode VBR
    OverallBitRate      191032188
    FrameRate           60.000
    FrameCount          19
    IsStreamable        Yes
    Encoded_Application Lavf60.16.100
    Encoded_Library     Lavf60.16.100
    ErrorDetectionType  Per level 1
Video
    Format              AV1
    Format_Profile      Main
    Format_Level        4.1
    CodecID             V_AV1
    Duration            0.317000000
    Width               1920
    Height              1080
    Sampled_Width       1920
    Sampled_Height      1080
    PixelAspectRatio    1.000
    DisplayAspectRatio  1.778
    FrameRate_Mode      CFR
    FrameRate           60.000
    FrameCount          19
    ColorSpace          YUV
    ChromaSubsampling   4:2:0
    BitDepth            8
    Delay               0.000
    Default             No
    Forced              No
    colour_range        Limited
    colour_range_Source Stream
Audio (omitting definitely irrelevant stuff)
    Format              FLAC
    CodecID             A_FLAC
    Duration            0.384000000
    BitRate_Mode        VBR
    SamplingRate        48000
    SamplingCount       18432
    BitDepth            16
    Compression_Mode    Lossless
    Delay               0.000
    Delay_Source        Container
    Default             No
    Forced              No

Here's a picture of the loss I'm noticing. Upscaling the middle image with RealESRGAN in SD-WebUI-Forge doesn't result in the loss seen in the right. Since AV1 is not truly lossless in the first place, I'd prefer FFV1, but Enhancr throws an error for that because the color space isn't YUV (media info says RGBA).

I'm gonna be honest, I exported from Blender because I can't really figure out ffmpeg, which it looks like I'll have to do if I want something truly lossless.

[Bubblemint864]

If you want something truly lossless try: ffmpeg -i input -c:v ffv1 -pix_fmt (whatever pixel format and bit depth you want, e.g. yuv444p10le, 444=4:4:4 subsampling, 10le = 10 bit) output.mkv

So AV1 seems to cause black frames and you haven't managed to get FFV1 to work. I don't see anything wrong with the AV1 file besides it not being 10 bit (which, if you are using AV1, is highly recommended) but that shouldn't be the cause of the issue. Maybe try encoding a YUV FFV1 file using the command shown above and see if the issue still occurs?

Also it does seem like Real-ESRGAN is nuking quite a bit of detail so perhaps you should look into using a different model or maybe even a shader based scaler like ArtCNN.

[WeatherWonders]

So, png to ffv1 to png is causing most pixels to end up being slightly off when using yuv, even with a bit depth of 14. Not that big of a deal, but it's weird.

I tried Enhancr on that with ShuffleCUGAN and two ESRGAN models. It was unable to use lollypop, but the other two passed with the same blocky compression effect. Another implication this is video compression is that it's less significant on the first frame.

Same issue with frame alignment: it starts with a black frame and the last two are missing.