Wrong playback speed after repeated setMedia() call

[oomek]

Hi, I've stumbled across some weird issue recently. I have one instance of the Player and I change the url from the list with the following code:

    player.setNextMedia(nullptr, -1);
    player.setState(State::Stopped);
    player.waitFor(State::Stopped);
    player.setMedia(nullptr);
    player.setMedia(file_name);
    player.prepare(0);
    player.setLoop(999);
    player.setBackgroundColor(0, 0, 0, -1);
    player.waitFor(State::Paused);
    auto& codec = player.mediaInfo().video[0].codec;
    player.setVideoSurfaceSize(codec.width, codec.height);
    texture.create(codec.width, codec.height);
    sprite.setTexture(texture.getTexture(), true);
    player.setState(State::Playing);
    // player.setPlaybackRate(1.0);

The first video runs fine, but all the consecutive files run slower ( or faster, depending on the fps of the first video), even the sound pitch reflects that. My videos have all different resolutions and frame rates ie 29.97, 30.00, 59.94, 60.00

Is the order of commands correct, or am I doing something wrong here?

[oomek]

I've also noticed a memory leak when I call the commands above repeatedly.

[oomek]

I've already tried player.setVideoSurfaceSize(-1, -1) but it had no effect on both issues.

[wang-bin]

The code you pasted looks fine. Can you share your example so I can reproduce the issues?

[oomek]

Here is the code, thanks https://github.com/oomek/mdktest

[oomek]

It seems like mdk remembers the frame rate of the first loaded video and when I reload another one ie from 29.97 to 30.00 I get audio pitch changes and frame skips/doubling. Additionally as you can see I do not call renderVideo() on each frame, it's the only way of having a smooth playback of 30fps videos on 60hz without stuttering. It's a different issue not related to video reloading.

[wang-bin]

It seems like mdk remembers the frame rate of the first loaded video

No. It plays at video frame rate * playbackRate(). I've tested your example on Mac, but can't produce the frame rate issue.

Additionally as you can see I do not call renderVideo() on each frame, it's the only way of having a smooth playback of 30fps videos on 60hz without stuttering. It's a different issue not related to video reloading.

30fps video on 60hz display will draw twice each frame, I redraw everything instead of using previous result, so may stutter on rpi. If sfml supports custom event, then you can call renderVideo() in a correct fps like this https://github.com/wang-bin/mdk-examples/blob/master/SDL/sdlplay.cpp#L93

[oomek]

Please try those 2 videos, the pitch of the audio should be the same for both, but it's not, and depending on which is loaded first the other loaded is stuttering as it's playing at the wrong speed, at least on PI4. v_calle.mp4.txt v_mario.mp4.txt

As I mentioned in https://github.com/wang-bin/mdk-sdk/issues/16 I can't remove those lines because of segfaults. I'm going to analyze this callback example now.

[oomek]

Tried with callback, and it stutters unfortunately, like without frame_time_counter

[oomek]

I forgot to mention. I've built sfml-pi with -DSFML_DRM I'm in KMS, not X11 environment.

[wang-bin]

Tried with callback, and it stutters unfortunately, like without frame_time_counter

Can you update your example? When receive the event, you should renderVideo() on the texture without displaying the texture on the window.

[wang-bin]

I tested your videos on my Mac, the speed is right. You can set env var MDK_LOG=1 and MDK_LOG_STATUS=1 to see the playback progress

[oomek]

log1.txt log2.txt

[wang-bin]

I mean you can check whether the playback progress increases correctly. Don't play fullscreen, so you can see the log when playing.

[oomek]

The timer is off on the second video too fast or too slow, depending on the order

[oomek]

Do you sync to audio by any chance? I saw something off about the audio in the log

[wang-bin]

Sync to audio if possible, sync to system clock if no audio track or no audio device

[oomek]

I'm out of ideas, tried everything...

[oomek]

Ok, I've figured this out. It's the encoded audio sample rate changes that is affecting the audio pitch and the video playback of the second loaded video. When I set player.setPlaybackRate(44100.0/48000.0) the second video that was choppy before has smooth playback. When I re-encoded the audio on both videos to match 48Khz there are no issues. It only manifests when I load 48kHz then 44.1kHz audio track or in reversed order.

[oomek]

Maybe that's gonna help: AudioBackendALSA ERROR@216>>> snd_pcm_drop(pcm_) (fffffffb): Input/output error same video played 1st AudioBackendALSA ERROR@285>>> snd_pcm_pause(pcm_, value) (ffffffb3): File descriptor in bad state same video played 2nd those are the only lines in a block that are different when the order of loading is swapped for those 2 videos with different audio sample rates.

[wang-bin]

I forgot to test whether input sample rate is supported. Will fix it this week.

[oomek]

I'm not sure if I understand. The video with 44.1kHz audio is playing fine when played 1st in a queue. What I believe happens is the audio sample rate switching fails for the consecutive files.

[oomek]

Have you managed to figure out how to fix that sample_rate sticking by any chance?

[wang-bin]

Have you managed to figure out how to fix that sample_rate sticking by any chance?

I guess enbling alsa resample will work. But my rpi supports all sample rates you mentioned via hdmi audio, so i can't test. What's your audio device? I can build a package for you to test

[oomek]

I mentioned it already. It's not a matter of supporting a certain sample rate. 44.1 and 48.0 play fine when played first in a queue. When I press left or right in my test app and the newly loaded file has a different samplerate then it plays audio and video at wrong speed.

[oomek]

Ok, I've found a workaround for the problem with incorrect sample rate switching. I got rid of the global instance of Player, used a pointer instead and I just recreate the instance in the load function like so:

if (player)
{
    player->setState(State::Stopped);
    player->waitFor(State::Stopped);
    delete player;
    player = NULL;
}
player = new Player();

[oomek]

Can't use it, it's stalling the main loop and I get stutter of other elements I animate. Even if I remove player->waitFor(State::Stopped);

[wang-bin]

I reproduce the issue on rpi4. Also tried glfwplay on rpi4, play v_mario.mp4 first, then drop v_calle.mp4 to glfwplay window, the speed becomes wrong(about 10% faster). But I can't reproduce it on macOS and windows. Can you try on desktop linux?

[oomek]

Also tried glfwplay on rpi4, play v_mario.mp4 first, then drop v_calle.mp4 to glfwplay window, the speed becomes wrong(about 10% faster)

Hmm, that could indicate driver issue, maybe we should report it on the Raspbian repo. Regarding tests on Linux: my other PC with Arch is currently set aside, I'm gonna try tomorrow.

[wang-bin]

can't reproduce on my linux computer