Subtitles get out of sync when resuming a movie, or the audio/video is out of sync with the subtitles.

[joost00719]

Describe the bug

Reproduce steps:

1. Watch a movie until like 50% of it.
2. Notice that the subtitles are (probably) in sync.
3. Shut down jellyfin app.
4. Start Jellyfin again, and try to resume the movie.
5. Notice that the subtitles are out of sync. However this doesn't happen at every movie.

I also noticed something else odd, which could explain why this bug happens.

If I go to timestamp 1:00:00 on the androidtv client (chromecast), and I go to 1:00:00 on the Windows 10 client, it seems like the androidtv client is like 5-10 seconds behind the Windows 10 client. The subtitles are 100% exactly on the same timestamp, so those appear at the identical time if you sync the two clients up, but the video and audio on the androidtv client seems to be 5-10 seconds behind, which could explain the subtitle sync issue. Long story short, it looks like the audio/video get's out of sync with the time, but the subtitles are actually correct.

For me this is a big issue since I have trouble hearing voices with background noise, especially if it's not my native tongue.

If any more info is needed, I am able to (privately) provide a test-account to my JF server to a developer, with sftp access to my library.

Logs

[2024-04-08 13:03:05.430 +02:00] [INF] [74] Jellyfin.Api.Helpers.MediaInfoHelper: User policy for "joost". EnablePlaybackRemuxing: True EnableVideoPlaybackTranscoding: True EnableAudioPlaybackTranscoding: True
[2024-04-08 13:03:05.431 +02:00] [INF] [74] Jellyfin.Api.Helpers.MediaInfoHelper: StreamBuilder.BuildVideoItem( Profile="AndroidTV-libVLC", Path="/data/media/movies/Harry Potter and the Deathly Hallows Part 2 (2011)/Harry.Potter.and.the.Deathly.Hallows.Part.2.2011.1080p.10bit.BluRay.6CH.x265.HEVC-PSA.mkv", AudioStreamIndex=null, SubtitleStreamIndex=null ) => ( PlayMethod=DirectPlay, TranscodeReason=0 ) "media:/videos/7a35d69c-9514-7a12-6085-859f3e9f0d8b/stream.mkv?MediaSourceId=7a35d69c95147a126085859f3e9f0d8b&Static=true&VideoCodec=hevc&AudioCodec=aac&AudioStreamIndex=2&api_key=<token>&Tag=bca4acb4a71f9ec7e60db48d1994d1f6"
[2024-04-08 13:03:05.581 +02:00] [INF] [36] Jellyfin.Api.Helpers.MediaInfoHelper: User policy for "joost". EnablePlaybackRemuxing: True EnableVideoPlaybackTranscoding: True EnableAudioPlaybackTranscoding: True
[2024-04-08 13:03:05.581 +02:00] [INF] [36] Jellyfin.Api.Helpers.MediaInfoHelper: StreamBuilder.BuildVideoItem( Profile="AndroidTV-ExoPlayer", Path="/data/media/movies/Harry Potter and the Deathly Hallows Part 2 (2011)/Harry.Potter.and.the.Deathly.Hallows.Part.2.2011.1080p.10bit.BluRay.6CH.x265.HEVC-PSA.mkv", AudioStreamIndex=null, SubtitleStreamIndex=null ) => ( PlayMethod=Transcode, TranscodeReason=AudioChannelsNotSupported ) "media:/videos/7a35d69c-9514-7a12-6085-859f3e9f0d8b/master.m3u8?MediaSourceId=7a35d69c95147a126085859f3e9f0d8b&VideoCodec=hevc,h264,hevc&AudioCodec=aac&AudioStreamIndex=2&VideoBitrate=99616000&AudioBitrate=384000&AudioSampleRate=48000&MaxFramerate=23.976025&api_key=<token>&RequireAvc=false&Tag=bca4acb4a71f9ec7e60db48d1994d1f6&SegmentContainer=ts&BreakOnNonKeyFrames=False&hevc-level=120&hevc-videobitdepth=10&hevc-profile=main10&hevc-audiochannels=6&aac-profile=he-aac&aac-audiochannels=2&TranscodeReasons=AudioChannelsNotSupported"
[2024-04-08 13:03:05.955 +02:00] [INF] [36] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: Adding playback tracker : 2e352282e0d23b46b66a01bd96896342f814fbbe-342c3b788c2e49dfb6226644434d66ef-7a35d69c95147a126085859f3e9f0d8b
[2024-04-08 13:03:05.955 +02:00] [INF] [36] Jellyfin.Plugin.PlaybackReporting.Data.PlaybackTracker: PlaybackTracker : Adding Start Event : 04/08/2024 13:03:05
[2024-04-08 13:03:05.955 +02:00] [INF] [36] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: Creating StartPlaybackTimer Task
[2024-04-08 13:03:05.955 +02:00] [INF] [36] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : Entered
[2024-04-08 13:03:06.836 +02:00] [INF] [73] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: Processing playback tracker : "2e352282e0d23b46b66a01bd96896342f814fbbe-342c3b788c2e49dfb6226644434d66ef-7a35d69c95147a126085859f3e9f0d8b"
[2024-04-08 13:03:07.946 +02:00] [INF] [56] Jellyfin.Api.Helpers.TranscodingJobHelper: "/usr/lib/jellyfin-ffmpeg/ffmpeg" "-analyzeduration 200M -fflags +genpts  -i file:\"/data/media/movies/Harry Potter and the Deathly Hallows Part 2 (2011)/Harry.Potter.and.the.Deathly.Hallows.Part.2.2011.1080p.10bit.BluRay.6CH.x265.HEVC-PSA.mkv\" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 copy -tag:v:0 hvc1 -bsf:v hevc_mp4toannexb -start_at_zero -codec:a:0 libfdk_aac -ac 2 -ab 384000 -ar 48000 -af \"volume=2\" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 6 -hls_segment_type mpegts -start_number 0 -hls_segment_filename \"/config/data/transcodes/b0c2898c3a5c96e5cb336ce3ed084331%d.ts\" -hls_playlist_type vod -hls_list_size 0 -y \"/config/data/transcodes/b0c2898c3a5c96e5cb336ce3ed084331.m3u8\""
[2024-04-08 13:03:09.122 +02:00] [INF] [69] Jellyfin.Api.Helpers.TranscodingJobHelper: Stopping ffmpeg process with q command for "/config/data/transcodes/b0c2898c3a5c96e5cb336ce3ed084331.m3u8"
[2024-04-08 13:03:09.189 +02:00] [INF] [69] Jellyfin.Api.Helpers.TranscodingJobHelper: FFmpeg exited with code 0
[2024-04-08 13:03:09.190 +02:00] [INF] [69] Jellyfin.Api.Helpers.TranscodingJobHelper: "/usr/lib/jellyfin-ffmpeg/ffmpeg" "-analyzeduration 200M -ss 01:18:16.567 -fflags +genpts  -i file:\"/data/media/movies/Harry Potter and the Deathly Hallows Part 2 (2011)/Harry.Potter.and.the.Deathly.Hallows.Part.2.2011.1080p.10bit.BluRay.6CH.x265.HEVC-PSA.mkv\" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 copy -tag:v:0 hvc1 -bsf:v hevc_mp4toannexb -start_at_zero -codec:a:0 libfdk_aac -ac 2 -ab 384000 -ar 48000 -af \"volume=2\" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 6 -hls_segment_type mpegts -start_number 782 -hls_segment_filename \"/config/data/transcodes/b0c2898c3a5c96e5cb336ce3ed084331%d.ts\" -hls_playlist_type vod -hls_list_size 0 -y \"/config/data/transcodes/b0c2898c3a5c96e5cb336ce3ed084331.m3u8\""
[2024-04-08 13:03:25.954 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: session.RemoteEndPoint : "192.168.1.254"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : event_playing_id     = "7a35d69c95147a126085859f3e9f0d8b"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : event_user_id        = "342c3b788c2e49dfb6226644434d66ef"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : event_user_id_int    = 1
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : session_playing_id   = "7a35d69c95147a126085859f3e9f0d8b"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : session_user_id      = "342c3b788c2e49dfb6226644434d66ef"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : play_method          = "Transcode (v:direct a:aac)"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : e.ClientName         = "Android TV"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : e.DeviceName         = "Joost"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : ItemName             = "Harry Potter and the Deathly Hallows: Part 2"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : ItemId               = "7a35d69c95147a126085859f3e9f0d8b"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : ItemType             = "Movie"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : All matches, playback registered
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: Playback tracker found, adding playback info : "2e352282e0d23b46b66a01bd96896342f814fbbe-342c3b788c2e49dfb6226644434d66ef-7a35d69c95147a126085859f3e9f0d8b"
[2024-04-08 13:03:25.955 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: Saving playback tracking activity in DB
[2024-04-08 13:03:25.977 +02:00] [INF] [65] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: StartPlaybackTimer : Exited
[2024-04-08 13:03:27.078 +02:00] [INF] [36] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: Processing playback tracker : "2e352282e0d23b46b66a01bd96896342f814fbbe-342c3b788c2e49dfb6226644434d66ef-7a35d69c95147a126085859f3e9f0d8b"
[2024-04-08 13:03:49.305 +02:00] [INF] [73] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: Processing playback tracker : "2e352282e0d23b46b66a01bd96896342f814fbbe-342c3b788c2e49dfb6226644434d66ef-7a35d69c95147a126085859f3e9f0d8b"

Application version

jellfyin-androidtv 0.16.7 release

Where did you install the app from?

Google Play

Device information

Chromecast HD

Android version

Android TV 12 (STTL.240206.002)

Jellyfin server version

10.8.13 (docker)

[VampiricAlien]

Any chance the media is being transcoded?

[joost00719]

Iirc it was a direct stream. But I'll be double checking later today and report back.

---

From: VampiricAlien @.> Sent: Tuesday, April 9, 2024 2:51:12 PM To: jellyfin/jellyfin-androidtv @.> Cc: Joost R. @.>; Author @.> Subject: Re: [jellyfin/jellyfin-androidtv] Subtitles get out of sync when resuming a movie, or the audio/video is out of sync with the subtitles. (Issue #3492)

Any chance the media is being transcoded?

— Reply to this email directly, view it on GitHubhttps://github.com/jellyfin/jellyfin-androidtv/issues/3492#issuecomment-2045110497, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFAMG55BKC2YSBVHNXHAFE3Y4PP4BAVCNFSM6AAAAABF4PM5KCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBVGEYTANBZG4. You are receiving this because you authored the thread.Message ID: @.***>

[joost00719]

Sorry, something came up yesterday. Anyways, it isn't being transcoded:

However, I tried to force transcoding by putting the quality to 1Mbit/s, and this "fixes" the sync issues. "fixes" in quotes because it makes the movie look bad obviously.

Is there any other way to force encoding, so I can temp-fix this until a proper fix is out?

[VampiricAlien]

In playback settings, Hardware encoding options might be a temp fix. Your CPU supports HEVC encoding.

[joost00719]

Hi VampiricAlien,

Thanks for the suggestion, however it did not temp-fix my issue. Are there other ways to force transcoding without losing resolution / on higher bitrates?

In case it is important, my setup is like this: i5 9500T running Proxmox Proxmox has a Debian VM with iGPU pass-through Debian vm runs Docker with GPU device mapping

And this are my JF settings for hardware decoding

[VampiricAlien]

In the app, menu, playback, under video there should be max streaming bitrate, set that to highest but don't select auto.

Can you post the video details such as: Video Title: 4K - HEVC - SDR Codec: HEVC Profile: Main 10 Level: 156 Resolution: 3840x2160 Aspect ratio: 16:9 Interlaced: No Framerate: 23.976025 Bitrate: 6072 kbps Bit depth: 10 bit Video range: SDR Video range type: SDR Color space: bt709 Color transfer: bt709 Color primaries: bt709 Pixel format: yuv420p10le Ref frames: 1 That's just an example. and also the device the client is using if you can.

[joost00719]

I've set it manually to 120mbps, but it's still direct stream and out of sync

I didn't know how to get that video detail in the format how you have it, I hope this screenshot is enough: If it's not, please let me know how I can get more information.

edit: Just noticed the last property is cut off in the last screenshot. It was "Ref frames" with value "1"

[VampiricAlien]

Silly question but are you sure the subtitle are correct? Have you tested on Jellyfin web, VLC, android client?

[joost00719]

Yes, on my phone and on my pc using the JellyFin windows client the subtitles are correct.

[VampiricAlien]

Have you looked at the option Allow subtitle extraction on the fly under playback?

[joost00719]

I haven't seen that option anywhere yet, but I'll see if I can find it tomorrow, and report back. Thanks for the suggestion.

[VampiricAlien]

You'll find the option in Jellyfin web.

[joost00719]

I just enabled it, rebooted JF and tried it again without luck. I think what I said in the original post is what is actually happening, which is the video/audio getting out of sync, but the subtitles actually being "correct".

windows: Chromecast:

Notice that both frames are at 35:30, have a different scene, but the exact same subtitles. This confirms for me that the subtitles are actually correct, but the video itself is somewhere else than JF expects, resulting in a sync issue between video and subtitles.

[VampiricAlien]

My last suggest would be to create a new audio track. ffmpeg -i inputfile.mkv -map 0:a:0 -ac 2 outputfile.aac ffmpeg -i input.mp4 -i outputfile.aac -map 0:v -map 0:s -map 1:a -c:v copy -c:s copy -c:a aac -strict experimental movie.mp4 Something like that or you can use a program to create a new 2 channel audio track. This doesn't fix the issue with the app but does better help you.

[joost00719]

I'll be waiting on a fix for now. And in the mean time I'll use a nuc to watch movies from the TV.

Thanks for helping and let me know if I can help in any way (e.g. Test a beta version)

[Deagarys]

I've been having this issue aswell, only happens on androidTv client. Subtitles that burn in are still correct even after resuming the movie/serie.

This started happening for me a few months ago, and playing the movie back from start does not seem to fix it. Hopefully it can be resolved in a future update of the androidTv client app.

[hcharbonnier]

In the Android App (or Web client), in the Subtiltes setting menu, you have an option to force burning all the subtitles in the video. Then restart the AndroidTV app and all your subtiltes will be burned in the video via transcoding. It is a pity this setting is not directly avilable in the AndroidTV app...

[joost00719]

Thanks @hcharbonnier . However I don't know which setting you mean:

Could you clarify which option exactly I need to select?

[hcharbonnier]

It is not "Subtitle mode"but the parameter right after. I don't know it's name in english but it should be something like "Burn the subtitles"

PS: I have a different value in the Android app and the Jellyfin web interface. I believe (but i'm not sure) you should modify this value from the website and not the android app if you want it to work in the androidTV app.

[joost00719]

I only seem to have these options.

However I checked via my web browser, instead of the Jellyfin Windows App, and there the setting is.

I'll be testing it out after work and update this message, and if it works, I'll also update the original post so people who are facing this issue know how to fix it without going through all the dialogue. Thank you.

[joost00719]

Just tested it with the "Burn subtitles" option set to "All". After that I rebooted the chrome cast, rebooted Jellyfin server and tried again. Unfortunately, it did not work. It doesn't seem like the subtitles are getting burned in correctly, even though the menu suggests it's global for that user, and not for the client since it doesn't say "These settings affect subtitles on this device" on that part.

I tried to test it by lowering the quality to the lowest setting possible and see if it affected the quality. In the browser it works, and ffmpeg is being shown in intel_gpu_top utility, in the Windows app and in the AndroidTv app it did not work and it is a direct play.

Is there any other way I can force subtitle burn in and/or transcoding without sacraficing bitrate/quality?

[hcharbonnier]

I don't really know, each time I modify this setting it take times before it became effective, and I don't konw why.

[joost00719]

I don't really know, each time I modify this setting it take times before it became effective, and I don't konw why.

I waited 2 days, and it did not become effective.

[hcharbonnier]

hum.. Maybe try to set this setting from the Android app instead of Jellyfin Web interface.

[joost00719]

Just checked in the JF discord server, and according to "EUPHIE IS SOOOO CUUUUUUUTE" (nice name btw), the option is only client-sided, which explains why the option does not do anything on the AndroidTV app.

https://discord.com/channels/772232779534172171/772247115681890314/1232034632296562839

[pierfied]

I've been digging into this a little bit and it would seem that the problem is due to exoplayer returning an incorrect value for the position when jellyfin calls getCurrentPosition(). This would explain why the subtitles are shown at the right time relative to the time according to the player but the video appears to be out of sync. It's possible this could be related to the following exoplayer issue: https://github.com/androidx/media/issues/356

[joost00719]

@pierfied that looks exactly like the issue I'm facing. Thanks for figuring out that it's actually unrelated to JF.

Would it be possible to get an option in the androidTV app menu to burn in subtitles? Maybe it's not the "technically correct" fix, but that would fix the sync issue.

[pierfied]

@joost00719 this is actually my first time ever looking at the JF codebase as I just wanted to try to dig into this exact issue. So, unfortunately, I don't think I'm the right person to tackle adding an option for burning in subtitles, but I'm sure another developer here may be able to.

However, I have been playing a bit with figuring out how to get around this particular issue. As far as I can tell, using the HLS protocol (as JF + exoplayer natively tries to do) results in exoplayer incorrectly adjusting the timestamps due to to how the JF server is returning TS segments. The JF client is requesting 6 second segments, but the returned segments contain closer to 10 seconds worth of frames for reasons that are beyond my level of knowledge (I think it has something to do with wanting to start each segment with a keyframe). As far as I can tell, exoplayer assumes that the first frame in the segment should correspond to the beginning of the 6 second segment even though that may not actually be the case. The problem then occurs when exoplayer adjusts all timestamps to force that first frame to fall on the beginning of the 6 second segment, resulting in incorrect timestamps for each frame.

Now, I've been trying some stuff to get around this issue and one workaround I've been playing with is simply disabling HLS playback with exoplayer and using a similar transcoding profile as the libVLC-based player: https://github.com/jellyfin/jellyfin-androidtv/compare/master...pierfied:jellyfin-androidtv:disable-hls

While I don't think this is optimal by any mean, it seems to be working for some limited testing I've done. I'm sure there are some consequences of disabling the HLS protocol that I am unaware of. As I am by no means an expert here and my knowledge of the various protocols, containers, and codecs is very limited, I would greatly appreciate any insights here from more experienced developers on this one.

[o-pteron]

I have exactly the same issue

[sotiris-bos]

I also have the same issue. And using libvlc causes micro-stuttering making jellyfin almost unusable on Android TV.

Maybe add some "Disable HLS", "Force Burn subtitles" options on the Android TV client app as a quick workaround?

[aculotarpa]

I have this very same problem not only when I stop and restart a video, but also when I jump back and forward during playback. The subtitles timing restart from zero. v. 0.16.11

[pierfied]

It seems like this particular issue also shows up across other JF clients and I think this is a server-side issue with how HLS is handled. Disabling HLS in the android tv client has worked well for me for a while now. I think adding an option in the settings to disable HLS playback might be a good solution until the HLS timestamp problem is properly diagnosed/fixed.

[viown]

It seems like this particular issue also shows up across other JF clients and I think this is a server-side issue with how HLS is handled. Disabling HLS in the android tv client has worked well for me for a while now. I think adding an option in the settings to disable HLS playback might be a good solution until the HLS timestamp problem is properly diagnosed/fixed.

@pierfied Hey, just wanted to ask if there are any issues that could arise from disabling HLS. I.e, would transcoding or direct streaming still work?

[pierfied]

It looks like this issue may be solved by https://github.com/jellyfin/jellyfin-androidtv/pull/3825.

[joost00719]

It looks like this issue may be solved by #3825.

Unfortunately I cannot test it because I have sold my Chrome Cast.

[nielsvanvelzen]

This is most likely fixed by #3825 yes. Closing as fixed.

[SkyDiver7199]

Not fixed. The subtitles work fine in my pc but when I watch same movie on Android TV, it goes out of sync. Works ok when I choose to play using external player. So something is wrong with Jellyfin

[nielsvanvelzen]

Did you try with 0.18?

[naphipps]

Definitely not fixed on Firefox. There is (and has been) a lot of issues with Jellyfin's support for Firefox. iPhone app works perfect. Debian 12 client works perfect. Fire stick works perfect. Chrome works perfect. Firefox seems to be getting the short stick lately.

[nielsvanvelzen]

How do you even use an Android TV app on Firefox??

[naphipps]

Are you asking for more information? I don't mind creating a new issue or doing a something myself if we're not going to make any progress on this issue.

[nielsvanvelzen]

If you encounter an issue with the Android TV app please do. But if it's not, you may want to look in the appropriate repository instead.

I'd like to remind everyone again that this issue is now closed because of changes made for the 0.18 release, which is not the current release but an upcoming version.

[naphipps]

Thanks. Will stay tuned.

If the issue exists in that release, I'm assuming to follow up here?

[maru801]

Thanks. Will stay tuned.

If the issue exists in that release, I'm assuming to follow up here?

You need to take this to the relevant github repo. This repo we're currently commenting in is dedicated to the Android TV app. It has nothing to do with Firefox.

The most appropriate repo would be jellyfin-web.

[naphipps]

Thanks. I'm assuming yes.