Video stuttering when rendering to SFML texture

[oomek]

I'm getting a very uneven framerate when rendering the video to SFML texure. When I play a 30fps video at 60Hz or 120Hz for example the callback function should fire every 2nd or 4th frame, shouldn't it? I've also tried putting all the calls from if (new_frame == true) inside the callback, but the result is the same. Am I using setRenderCallback() correctly?

Platform: Windows 11

#include <SFML/Window.hpp>
#include <SFML/Graphics.hpp>
#include "mdk/Player.h"
using namespace MDK_NS;

#include <chrono>
using namespace std::chrono;

#include <atomic>

int main(int argc, char** argv)
{
    sf::Context vid_context;
    vid_context.setActive(true);

    Player player;

    std::atomic<int> new_frame(0);
    auto t0 = steady_clock::now();

    player.setRenderCallback([&](void*)
    {
        new_frame++;
        const auto t = steady_clock::now();
        printf("elapsed: %lld new_frame: %d\n", duration_cast<milliseconds>(t-t0).count(), new_frame.load());
        t0 = t;
    });

    player.setMedia(argv[argc-1]);
    player.prepare();

    for(;;)
    {
        if (player.mediaStatus() > MediaStatus::Buffering) break;
        if (player.mediaStatus() == MediaStatus::Invalid) return 0;
    }

    auto video_info = player.mediaInfo().video[0];
    player.setVideoSurfaceSize(video_info.codec.width * video_info.codec.par, video_info.codec.height);

    sf::RenderWindow window(sf::VideoMode(800, 600), "Test");
    window.setVerticalSyncEnabled(true);

    sf::RenderTexture texture;

    if (!texture.create(video_info.codec.width * video_info.codec.par, video_info.codec.height))
        return -1;

    player.set(State::Playing);

    while (window.isOpen())
    {
        sf::Event event;
        while (window.pollEvent(event))
        {
            if (event.type == sf::Event::Closed)
                window.close();
            if (event.type == sf::Event::KeyPressed)
            {
                if (event.key.code == sf::Keyboard::Escape)
                {
                    player.setVideoSurfaceSize(-1, -1);
                    return 0;
                }
            }
        }

        if (new_frame > 0)
        {
            vid_context.setActive(true);
            texture.setActive(true);
            player.renderVideo();
            texture.setActive(false);
            texture.display();
            vid_context.setActive(false);
            new_frame = 0;
        }

        window.setActive(true);
        window.clear();
        sf::Sprite sprite(texture.getTexture());
        window.draw(sprite);
        window.display();
    }
    return 0;
}

[wang-bin]

you can print the draw interval

[oomek]

I would need a little more help please.

[wang-bin]

if (new_frame == true) {
        const auto t = steady_clock::now();
        printf("elapsed: %lld\n", duration_cast<milliseconds>(t-t0).count());
        t0 = t;

[oomek]

Playback of 30fps video at 60Hz

elapsed: 1
elapsed: 71
elapsed: 16
elapsed: 50
elapsed: 16
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 34
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 49
elapsed: 34
elapsed: 16
elapsed: 32
elapsed: 49
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 49
elapsed: 16
elapsed: 49
elapsed: 16
elapsed: 33
elapsed: 33
elapsed: 16
elapsed: 49
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 50
elapsed: 16
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 50
elapsed: 17
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 50
elapsed: 16
elapsed: 31
elapsed: 33

[oomek]

Playback of 60fps video at 60Hz

elapsed: 1
elapsed: 76
elapsed: 33
elapsed: 33
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 33
elapsed: 17
elapsed: 17
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 33
elapsed: 16
elapsed: 31
elapsed: 33
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 17
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 17
elapsed: 16
elapsed: 33
elapsed: 31
elapsed: 33
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 16
elapsed: 16
elapsed: 33
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 33
elapsed: 33
elapsed: 33
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 33
elapsed: 16

[oomek]

Elapsed print after window.display()

elapsed: 48
elapsed: 5
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16
elapsed: 16

[oomek]

Video1

Video: MPEG4 Video (H264) 854x480 30fps 2025kbps [V: VideoHandler [eng] (h264 main L4.1, yuv420p, 854x480, 2025 kb/s)]
Audio: AAC 44100Hz stereo 126kbps [A: Stereo [eng] (aac lc, 44100 Hz, stereo, 126 kb/s)]

Video2

Video: MPEG4 Video (H264) 640x480 59.94fps 375kbps [V: h264 high L3.1, yuv420p, 640x480, 375 kb/s]
Audio: AAC 48000Hz stereo 64kbps [A: aac lc, 48000 Hz, stereo, 64 kb/s]

[wang-bin]

what about the interval in render callback? what about changing new_frame type to atomic and check if (new_frame > 0)?

[oomek]

I've updated the code in my first post

elapsed: 30 new_frame: 1
elapsed: 156 new_frame: 2
elapsed: 2 new_frame: 3
elapsed: 1 new_frame: 4
elapsed: 41 new_frame: 1
elapsed: 2 new_frame: 2
elapsed: 43 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 62 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 15 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 47 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 29 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 29 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 29 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 48 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 29 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 15 new_frame: 1
elapsed: 47 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 47 new_frame: 1
elapsed: 15 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 47 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 45 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 32 new_frame: 1
elapsed: 32 new_frame: 1
elapsed: 15 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 45 new_frame: 1
elapsed: 15 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 45 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1

[oomek]

I've noticed that removing sf::Context and switching Threaded Optimization in nvidiaProfileInspector to ON has an effect on player.renderVideo() which is stalling the thread for 16ms. Unfortunately it has no effect on the stutter.

[oomek]

But with Threaded Optimization set to ON the log looks different: Btw, In my project where I use pure ffmpeg to play videos I need to disable Threaded Optimization because I get stuttering.

elapsed: 29 new_frame: 1
elapsed: 164 new_frame: 2
elapsed: 29 new_frame: 3
elapsed: 46 new_frame: 1
elapsed: 2 new_frame: 2
elapsed: 28 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 2
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 47 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 45 new_frame: 1
elapsed: 15 new_frame: 2
elapsed: 46 new_frame: 1
elapsed: 29 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 32 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 2
elapsed: 31 new_frame: 1
elapsed: 47 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 15 new_frame: 2
elapsed: 46 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 2
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 32 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 15 new_frame: 2
elapsed: 46 new_frame: 1
elapsed: 30 new_frame: 2
elapsed: 31 new_frame: 1
elapsed: 45 new_frame: 1
elapsed: 29 new_frame: 1
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 31 new_frame: 2
elapsed: 31 new_frame: 1
elapsed: 30 new_frame: 1
elapsed: 46 new_frame: 1
elapsed: 15 new_frame: 2

[oomek]

With the following hardcoded framerate example I was able to render 30fps@60Hz smoothly. One condition Threaded Optimization must be OFF, otherwise there is occasional stutter and renderVideo() takes 16ms

#include <SFML/Window.hpp>
#include <SFML/Graphics.hpp>
#include "mdk/Player.h"
using namespace MDK_NS;

#include <iostream>

int main(int argc, char** argv)
{
    int flipflop = 1;

    Player player;

    player.setMedia(argv[argc-1]);
    player.prepare();

    for(;;)
    {
        if (player.mediaStatus() > MediaStatus::Buffering) break;
        if (player.mediaStatus() == MediaStatus::Invalid) return 0;
    }

    auto video_info = player.mediaInfo().video[0];
    player.setVideoSurfaceSize(video_info.codec.width * video_info.codec.par, video_info.codec.height);

    sf::RenderWindow window(sf::VideoMode(800, 600), "Test");
    window.setVerticalSyncEnabled(true);

    sf::RenderTexture texture;

    if (!texture.create(video_info.codec.width * video_info.codec.par, video_info.codec.height))
        return -1;

    player.set(State::Playing);

    while (window.isOpen())
    {
        sf::Event event;
        while (window.pollEvent(event))
        {
            if (event.type == sf::Event::Closed)
                window.close();
            if (event.type == sf::Event::KeyPressed)
            {
                if (event.key.code == sf::Keyboard::Escape)
                {
                    player.setVideoSurfaceSize(-1, -1);
                    return 0;
                }
            }
        }

        if (flipflop == 1)
        {
            texture.setActive(true);
            sf::Clock clk;
            player.renderVideo();
            std::cout << clk.getElapsedTime().asMicroseconds() << std::endl;
            texture.display();
            texture.setActive(false);
        }

        flipflop *= -1;

        window.setActive(true);
        window.clear();
        sf::Sprite sprite(texture.getTexture());
        window.draw(sprite);
        window.display();
    }
    return 0;
}

[oomek]

The following code is stuttering very little with TO-OFF, Can this be achieved with callbacks?

#include <SFML/Window.hpp>
#include <SFML/Graphics.hpp>
#include "mdk/Player.h"
using namespace MDK_NS;

#include <iostream>

int main(int argc, char** argv)
{
    sf::Context ctx;
    Player player;

    player.setMedia(argv[argc-1]);
    player.prepare();

    for(;;)
    {
        if (player.mediaStatus() > MediaStatus::Buffering) break;
        if (player.mediaStatus() == MediaStatus::Invalid) return 0;
    }

    auto video_info = player.mediaInfo().video[0];
    player.setVideoSurfaceSize(video_info.codec.width * video_info.codec.par, video_info.codec.height);

    float fps = video_info.codec.frame_rate;
    std::cout << video_info.codec.frame_rate << std::endl;

    ctx.setActive(false);

    sf::RenderWindow window(sf::VideoMode(640, 480), "Test");
    window.setVerticalSyncEnabled(true);

    sf::RenderTexture texture;

    if (!texture.create(video_info.codec.width * video_info.codec.par, video_info.codec.height))
        return -1;

    player.set(State::Playing);

    sf::Clock clock;
    sf::Time timeSinceLastUpdate = sf::Time::Zero;
    sf::Time timePerFrame = sf::seconds(1.0f / fps);

    while (window.isOpen())
    {
        sf::Event event;
        while (window.pollEvent(event))
        {
            if (event.type == sf::Event::Closed)
                window.close();
            if (event.type == sf::Event::KeyPressed)
            {
                if (event.key.code == sf::Keyboard::Escape)
                {
                    player.setVideoSurfaceSize(-1, -1);
                    return 0;
                }
            }
        }

        timeSinceLastUpdate += clock.restart();

        if (timeSinceLastUpdate > timePerFrame)
        {
            ctx.setActive(true);
            texture.setActive(true);
            // sf::Clock c;
            player.renderVideo();
            // std::cout << c.getElapsedTime().asMicroseconds() << std::endl;
            texture.display();
            texture.setActive(false);
            ctx.setActive(false);

            timeSinceLastUpdate -= timePerFrame;
        }

        window.setActive(true);
        window.clear();
        sf::Sprite sprite(texture.getTexture());
        window.draw(sprite);
        window.display();
    }
    return 0;
}

https://github.com/wang-bin/mdk-sdk/assets/2974860/28d4b8f6-9e53-45f7-99c5-0feca9cf95b6

https://github.com/wang-bin/mdk-sdk/assets/2974860/609695de-01fa-4304-97e1-326eef4e5033

I've attached test videos I'm using

[wang-bin]

player.setFrameRate(fps)

[oomek]

Thank you for sharing this undocumented function. This definitely helps in the situation when for example the framerate is 29.97 and the refresh rate of the monitor is 59.97. Unfortunately the hassle with calling renderVideo() at a certain moment persists. I'm still unable to call

texture.setActive(true);
player.renderVideo();
texture.display();
texture.setActive(false);

on each frame, because that is causing a massive judder.

Would it be possible to make renderVideo() call with some parameter that would just tell it internally to return without updating the texture if the new frame is not ready to be scheduled for display?

[oomek]

It appears that this juddering is not caused by my implementation. I tried glfwplay.exe and it's even worse. Would you be able to investigate and see if it can be fixed please?

Below is a screen capture of a smooth 60fps video posted above played using glfwplay.exe https://github.com/wang-bin/mdk-sdk/assets/2974860/70ebb07c-4200-4589-9bb6-0a764065d18e

[oomek]

So, is this solvable?

[wang-bin]

Would it be possible to make renderVideo() call with some parameter that would just tell it internally to return without updating the texture if the new frame is not ready to be scheduled for display?

render callback is invoked when a frame is ready, then call renderVideo() by user

It appears that this juddering is not caused by my implementation. I tried glfwplay.exe and it's even worse. Would you be able to investigate and see if it can be fixed please?

depending on video frame rate and display frame rate

[oomek]

This judder is weird. It skips frames and repeats at the same time, check the video I posted frame by frame. For a framerate mismatch it should only do one or the other thing, not both. Seems like a timing issue to me.

[oomek]

This works almost perfect, a little frame skip once for a while, but no forward and back judder. I think the videoRender callback should be fixed.

#include <SFML/Window.hpp>
#include <SFML/Graphics.hpp>
#include "mdk/Player.h"
using namespace MDK_NS;
using namespace std;

#include <iostream>
#include <cmath>

int main(int argc, char** argv)
{
    Player player;

    player.setMedia(argv[argc-1]);
    player.prepare();

    for(;;)
    {
        if (player.mediaStatus() > MediaStatus::Buffering) break;
        if (player.mediaStatus() == MediaStatus::Invalid) return 0;
    }

    auto video_info = player.mediaInfo().video[0];
    player.setVideoSurfaceSize(video_info.codec.width * video_info.codec.par, video_info.codec.height);

    float fps = video_info.codec.frame_rate;

    sf::RenderWindow window(sf::VideoMode(640, 480), "Test");
    window.setVerticalSyncEnabled(true);

    sf::RenderTexture texture;

    if (!texture.create(video_info.codec.width * video_info.codec.par, video_info.codec.height))
        return -1;

    player.set(State::Playing);
    // player.setFrameRate(29);

    sf::Clock clock;
    sf::Time timeSinceLastFrame = sf::Time::Zero;
    sf::Time timePerFrame = sf::seconds(1.0 / fps);

    float framerate = 60.0f;

    sf::Font font;
    if (!font.loadFromFile("RobotoMono-Regular.ttf")){cout << "Font not found." << endl;}

    sf::Text fps_text;
    fps_text.setFont(font);
    fps_text.setString(to_string(fps));
    fps_text.setCharacterSize(24);

    sf::Text framerate_text;
    framerate_text.setFont(font);
    framerate_text.setString(to_string(framerate));
    framerate_text.setCharacterSize(24);
    framerate_text.setPosition(0, floor(fps_text.getLocalBounds().height * 1.25));

    sf::Clock framerate_clock;
    sf::Time next_timestamp = sf::seconds(0);

    sf::Clock video_timer;

    while (window.isOpen())
    {
        sf::Event event;
        while (window.pollEvent(event))
        {
            if (event.type == sf::Event::Closed)
                window.close();
            if (event.type == sf::Event::KeyPressed)
            {
                if (event.key.code == sf::Keyboard::Escape)
                {
                    player.setVideoSurfaceSize(-1, -1);
                    return 0;
                }
            }
        }

        if ( next_timestamp < video_timer.getElapsedTime() - timePerFrame )
        {
            texture.setActive(true);
            sf::Time last_timestamp = sf::seconds(player.renderVideo());
            next_timestamp += timePerFrame;
            texture.display();
            texture.setActive(false);
        }

        window.setActive(true);
        window.clear();
        sf::Sprite sprite(texture.getTexture());
        window.draw(sprite);
        window.draw(fps_text);
        window.draw(framerate_text);
        window.display();

        sf::Time elapsedTime = framerate_clock.restart();

        if(elapsedTime.asMilliseconds() > 1.0f)
            framerate = framerate * 0.99f + 0.01f / elapsedTime.asSeconds();
        framerate_text.setString(to_string(framerate));
    }
    return 0;
}

[oomek]

I was trying to add if (next_timestamp < last_timestamp + timePerFrame) next_timestamp = last_timestamp + timePerFrame; after renderVideo(), but that caused a framerate drop to 1/4 of the actual video framerate.