[bug] Video playback offset after jumping to different playback position

[mariusoei]

Dear all,

first of all, thanks for all the work on this awesome app! I'm having a playback problem that creates an offset between video (including audio) and the subtitles. As far as I can tell it's being played back via DirectStream. I'm running Jellyfin 10.5.3 in a Docker container in my local network.

Edit: I'm using the Jellyfin App from the Google Play Store (not sure how to check the version) on a Galaxy S10+.

If I start the playback from the start, my srt subtitles (external file) are in sync with the video and everything's fine. However, if I jump to a different position in the video stream, the video and subtitles are out of sync (I just compared it with my computer, the difference in one case was 24 seconds, in the other 18s). Audio and video are in sync, but the playback position is wrong.

An example: I jump to playback position 1:00. The video starts to play (after about 15s) and the playback bar shows 1:00. The video playback however will actually be at position 1:24. The subtitles will show according to the playback bar at 1:00. So the subtitles lag behind by a lot.

Of course I would expect the video playback to actually be at the position shown in the playback bar.

This is the output of mediainfo for the file in question:

General
Complete name                            : ---
Format                                   : Matroska
Format version                           : Version 2
File size                                : 3.41 GiB
Duration                                 : 43 min 0 s
Overall bit rate                         : 11.4 Mb/s
Writing application                      : mkvmerge v6.6.0 ('The Edge Of The In Between') built on Dec 1 2013 17:55:00
Writing library                          : libebml v1.3.0 + libmatroska v1.4.1

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4.1
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 4 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 43 min 0 s
Bit rate                                 : 8 337 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 (23976/1000) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.168
Stream size                              : 2.44 GiB (71%)
Writing library                          : x264 core 148 r2638 7599210
Encoding settings                        : cabac=1 / ref=4 / deblock=1:-3:-3 / analyse=0x3:0x133 / me=umh / subme=8 / psy=1 / psy_rd=1.00:0.15 =1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=-3 / threads=12 / lookahead_thliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=2 / b_bias=0 / deightb=1 / open_gop=0 / weightp=2 / keyint=240 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=50 / rc=2pass / mbtree=1 / bitrattetol=1.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / cplxblur=20.0 / qblur=0.5 / ip_ratio=1.40 / aq=1:1.00
Language                                 : English
Default                                  : Yes
Forced                                   : No

Audio #1
ID                                       : 2
Format                                   : DTS
Format/Info                              : Digital Theater Systems
Codec ID                                 : A_DTS
Duration                                 : 43 min 0 s
Bit rate mode                            : Constant
Bit rate                                 : 1 509 kb/s
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 kHz
Frame rate                               : 93.750 FPS (512 SPF)
Bit depth                                : 24 bits
Compression mode                         : Lossy
Stream size                              : 464 MiB (13%)
Title                                    : DTS
Language                                 : German
Default                                  : Yes
Forced                                   : No

Audio #2
ID                                       : 3
Format                                   : DTS
Format/Info                              : Digital Theater Systems
Codec ID                                 : A_DTS
Duration                                 : 43 min 0 s
Bit rate mode                            : Constant
Bit rate                                 : 1 509 kb/s
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 kHz
Frame rate                               : 93.750 FPS (512 SPF)
Bit depth                                : 24 bits
Compression mode                         : Lossy
Stream size                              : 464 MiB (13%)
Title                                    : DTS
Language                                 : English
Default                                  : Yes
Forced                                   : No

Text
ID                                       : 4
Format                                   : UTF-8
Codec ID                                 : S_TEXT/UTF8
Codec ID/Info                            : UTF-8 Plain Text
Title                                    : Forced
Language                                 : German
Default                                  : Yes
Forced                                   : No

Menu
00:00:00.000                             : en:00:00:00.000
00:03:41.304                             : en:00:03:41.304
00:09:47.837                             : en:00:09:47.837
00:17:22.208                             : en:00:17:22.208
00:22:56.875                             : en:00:22:56.875
00:31:20.045                             : en:00:31:20.045
00:39:57.103                             : en:00:39:57.103

[Artiume]

An example: I jump to playback position 1:00. The video starts to play (after about 15s) and the playback bar shows 1:00. The video playback however will actually be at position 1:24. The subtitles will show according to the playback bar at 1:00. So the subtitles lag behind by a lot.

So is it the subtitles are lagging or the subtitles are correct and the video is off?

[mariusoei]

If we take the playback bar as the reference, the subtitles are correct and the video+audio is off/ahead.

[Artiume]

Would you be willing to dig into the subtitles file and make sure that the 1 minute mark for the subtitles and the 1 minute mark for the playback bar match? I think that'd be a good reference point

[mariusoei]

I already did :-) The times given in the subtitle file match the playback bar.

[Artiume]

Ok, so it seems like the subtitles are working as intended and the stream is out of sync. Is your video and audio transcoding or just the audio?

[mariusoei]

I guess this part of the ffmpeg log means that audio is transcoded, but video isn't?

Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:2 -> #0:1 (dts (dca) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
[hls @ 0x558f5b2df8c0] Opening '/config/data/transcodes/e7062e6aecdcd80a8f29424e76e3d309285.ts' for writing
Output #0, hls, to '/config/data/transcodes/e7062e6aecdcd80a8f29424e76e3d309.m3u8':
  Metadata:
    encoder         : Lavf58.29.100
    Stream #0:0: Video: h264 (High), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 23.98 fps, 23.98 tbr, 90k tbn, 23.98 tbc (default)
    Stream #0:1: Audio: mp3 (libmp3lame), 48000 Hz, stereo, fltp, 384 kb/s (default)

Also not sure if I mentioned it, but the offset is different every time and it seems like it depends on the time the stream takes to continue after skipping ahead. So if I skip ahead, sometimes the stream starts again almost instantly and I will have an offset of maybe a second. And another time I skip ahead and the stream takes 15s to start again - in that case, the offset was much larger.

So just maybe it could be that the video stream is "running" (albeit not being displayed) but the subtitles are waiting for playback to actually continue?

[Artiume]

Something like that. I think the problem is that the playback doesn't want to start playing on a non-keyframe. So when you do +30 seconds of a skip, when it skips there, it's waiting on a keyframe (or known as an I-frame) or something and since they can be spread apart, it glitches itself, waiting for that next keyframe.

Here's a good read off wiki to explain what's going on in a technical sense for the frames themselves https://en.m.wikipedia.org/wiki/Inter_frame

[mariusoei]

I read up a bit on that and I checked the I-frame positions in the video. At least for the first 2 minutes, they're never further apart then 10s, so I wonder if this can explain the offset of up to ~30s.

Also, I think I was wrong about the correlation between "hickup" length and offset. The offset is consistent when skipping to the same point, but seems to vary between 10 and 30s.

I also tried v10.5.0 (didn't display subtitles at all) and 10.4.3 (same offset).

[JustAMan]

Do you see any cases of files without subtitles having discrepancy between play bar and movie timings?

[Artiume]

I think vlc or another program would be good for comparing

[mariusoei]

I tried 2 files:

1. Different show without subtitles and in lower resolution

   
   General
   Unique ID                                : 199072188029289859600011968451014219297 (0x95C3E8A73FF37D4095ADA8B08576D621)
   Complete name                            : VideoWithoutSubtitles.mkv
   Format                                   : Matroska
   Format version                           : Version 4 / Version 2
   File size                                : 103 MiB
   Duration                                 : 18 min 14 s
   Overall bit rate                         : 789 kb/s
   Encoded date                             : UTC 2016-08-29 22:52:31
   Writing application                      : mkvmerge v7.9.0 ('Birds') 64bit
   Writing library                          : libebml v1.3.1 + libmatroska v1.4.2

Video ID : 1 Format : AVC Format/Info : Advanced Video Codec Format profile : High@L3.1 Format settings : CABAC / 8 Ref Frames Format settings, CABAC : Yes Format settings, ReFrames : 8 frames Codec ID : V_MPEG4/ISO/AVC Duration : 18 min 14 s Bit rate : 691 kb/s Width : 720 pixels Height : 400 pixels Display aspect ratio : 16:9 Frame rate mode : Constant Frame rate : 23.976 (24000/1001) FPS Color space : YUV Chroma subsampling : 4:2:0 Bit depth : 8 bits Scan type : Progressive Bits/(Pixel*Frame) : 0.100 Stream size : 90.2 MiB (88%) Writing library : x264 core 148 r2705 3f5ed56 Encoding settings : cabac=1 / ref=8 / deblock=1:0:0 / analyse=0x3:0x113 / me=umh / subme=9 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=6 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=5 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=240 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=60 / rc=crf / mbtree=1 / crf=21.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / aq=2:1.00 Default : Yes Forced : No

Audio ID : 2 Format : AAC Format/Info : Advanced Audio Codec Format profile : LC Codec ID : A_AAC-2 Duration : 18 min 14 s Bit rate : 96.0 kb/s Channel(s) : 2 channels Channel positions : Front: L R Sampling rate : 48.0 kHz Frame rate : 46.875 FPS (1024 SPF) Compression mode : Lossy Delay relative to video : 9 ms Stream size : 12.4 MiB (12%) Language : English Default : Yes Forced : No

Result: No offset between playback bar and video at multiple positions (checked with MPC-HC).

2. The original video: There were both .srt subtitles and an embedded subtitle stream in the mkv file.
Neither seemed to be the cause of the offset: I first removed the external subtitle files (same offset) and then also removed the subtitle stream using mkvmerge (same offset).

Looks like this isn't a subtitle issue at all.

[Artiume]

I want to say it only happens when you Direct Stream (convert audio only). Could you confirm?

[JustAMan]

Probably direct stream (remux). Basically not direct play and not full video transcode. Right?

[mariusoei]

I'll check today. There is probably no way to force DirectPlay instead of DirectStream, right? I know I can force transcoding.

[Artiume]

No, an easy codec to look for is 6 channel audio. It should have to convert the audio to 2 channel

[mariusoei]

I tried it with transcoding: No offset.

And one more thing I just noticed: I tried with a different show and in that case, the video is lagging behind. Meaning that the time offset was positive for one show and negative for the other.

[JustAMan]

@mariusoei can you check with mobile Firefox and some desktop browser please?

[mariusoei]

I apologize for the delay... On desktop, the problem doesn't appear at all. I tried with Firefox and Brave (Chromium).

On mobile browsers (again Firefox and Brave on Android) I can't get the video to play. On Firefox I get "There was an issue with the client profiling and the server isn't sending a compatible media format." On Brave, the video simply doesn't start playing, although I can't see anything suspicious in the logs.

[Artiume]

Are you looking at your ffmpeg logs? For all 4 scenarios, can you verify the type of Playback occurring? Direct Play ,Direct Stream, Transcoding.

On mobile browsers (again Firefox and Brave on Android) I can't get the video to play. On Firefox I get "There was an issue with the client profiling and the server isn't sending a compatible media format." On Brave, the video simply doesn't start playing, although I can't see anything suspicious in the logs.

Do you have transcoding disabled or anything? Kinda weird for it to not be working.

[mariusoei]

For desktop browsers it's always Direct Stream.

I put the log lines that appear when I try to play on a mobile browser in this gist: jellyfin.log

It doesn't create an ffmpeg log when I play on a mobile browser...

[Artiume]

Can you enable debug logging in jellyfin? To enable console for mobile browsers can be annoying. Might be easier to install Firefox ESR lol. I read back through the ticket but I didn't see it asked anywhere, do you have a baseUrl?

Also, I think I was wrong about the correlation between "hickup" length and offset. The offset is consistent when skipping to the same point, but seems to vary between 10 and 30s.

How often did you skip back and forth to see how consistent the correlation was? I think that the issue centers around the hls segments and video codec segments. The playback isn't able to properly keep track of the video position. You said the screen goes black for a few seconds, I wonder if it's blank due to waiting for the next available I-frame to be available.

Let's rename the ticket to Video Playback offset since the subtitles were a red herring.

[mariusoei]

So I tried jumping back and forth this time, the offset was not consistent. Sometimes positive, sometimes negative, always around 10-20s. It was consistent when skipping with double tap after starting the video.

Sometimes it would also get stuck after a few seconds of playback and then start again later on, wasn't able to reproduce this though.

I have a domain that jellyfin runs on behind a traefik reverse proxy. I haven't set the baseUrl parameter.

[Artiume]

The 10-20s, is it that far off even deep into the media? If so, maybe it's not a bad reference between the segments but the video is unable to enter without an I-frame and it skips to the next available i-frame.

[JustAMan]

@mariusoei please update to 10.5.4 before testing further, it has a few fixes specifically for mobile browsers (and a new version of Android app would have another fix, too, but it's not out yet).

[dkanada]

There's actually a fix for custom offset as well, maybe that will solve your issue.

[mariusoei]

@mariusoei please update to 10.5.4 before testing further, it has a few fixes specifically for mobile browsers (and a new version of Android app would have another fix, too, but it's not out yet).

I updated to 10.5.4, the problem persists (I don't see any difference to before).

How often did you skip back and forth to see how consistent the correlation was?

I skipped to the 1:00 mark from all over the video (at least 5 times), the offset was always the same as far as I can tell.

The 10-20s, is it that far off even deep into the media? If so, maybe it's not a bad reference between the segments but the video is unable to enter without an I-frame and it skips to the next available i-frame.

Yes, I got offsets of various lengths about 30-40 minutes in.

Can you enable debug logging in jellyfin?

I enabled debug logging in Jellyfin. Here are some logs when trying to play from mobile browsers and when playing and skipping a few times in the app: https://gist.github.com/mariusoei/b047d21134e0faa2ad08a0cc2aa9da45

Note that the info "Bitrate exceeds DirectStream limit: media bitrate: 14430870, max bitrate: 400000" only appears when playing from Brave (using the same user for every test in Jellyfin). I can't enable transcoding in the browsers either.

There's actually a fix for custom offset as well, maybe that will solve your issue.

Is this fix included in 10.5.4? Do I need to do anything to make use of it?

Do you have transcoding disabled or anything? Kinda weird for it to not be working.

Transcoding is enabled and working on my desktop browser.

[Artiume]

Probably related https://github.com/jellyfin/jellyfin-web/issues/879

[olskar]

I also have this issue on Android, tell me if i can help track down What is wrong somehow

[Artiume]

The root issue is inside how hls is handling the break on non-keyframes. We've been having discussions on how to fix it such as shifting to DASH. There isn't much that can be done atm :/

[olskar]

The root issue is inside how hls is handling the break on non-keyframes. We've been having discussions on how to fix it such as shifting to DASH. There isn't much that can be done atm :/

Hm, this does not seem to be an issue in Android tv client, only in Android client @Artiume?

[Artiume]

AndroidTV uses Progressive streaming and not hls streaming so it wouldn't affect AndroidTV. It's actually impacting not just Android but the majority of clients like ios, web, Android.