Inconsistent subtitle functionality with MP4VTT via url streaming

[JustinGuruTech]

Hey all, I'm trying to enable subtitles within an Android app using ExoPlayer for our video playback. We have videos stored at a url in our CMS, and they are in mp4 file format with embedded vtt subtitles (could be missing some info here, I don't know codecs super well). The subtitles will load, but are consistently delayed 10-20 seconds, at which point all the subtitles leading up to that point will quickly flash on the screen, after which the subtitles work as expected. Subtitles continue to work just fine when seeking forward, but seeking backwards causes the 10-20 second delay again. Additionally, I'm testing with both a 2:30 video and a 2:13 video. For the 2:13 video, the delay is consistently 10 seconds, whereas for the 2:30 video the delay is 20 seconds. Seeking backwards will sometimes increase the delay length. I loaded both videos into the demo app and am getting the same behavior in the demo app as in ours.

To load the videos into the demo app, I replaced the function at the bottom of PlayerActivity.java with the following to force my video to load. I then expand the subtitle section and press the first option (TTML positioning, not sure if it actually matters which one). Once the player loads, I go manually select the English text track for the video and the above behavior is replicated.

private static List<MediaItem> createMediaItems(Intent intent, DownloadTracker downloadTracker) {
    List<MediaItem> mediaItems = new ArrayList<>();
    MediaItem.Builder builder = MediaItem.EMPTY.buildUpon();
    builder
//        .setUri("https://drive.google.com/uc?export=download&id=1II826CYfICX54-IlHth8Z291jLsB20E1") // 2:13 video
        .setUri("https://drive.google.com/uc?export=download&id=1DMmP7bTrI4CAB_3Znrc9CmCrQ0ko8e1y") // 2:30 video
        .setMimeType(MimeTypes.APPLICATION_MP4VTT);
    mediaItems.add(builder.build());
    return mediaItems;
  }

Within our app, I'm initializing the player with a selection on the subtitle track like so

// initialize player with selections on subtitle track
val trackSelector = DefaultTrackSelector(context)
val trackSelectorParams = trackSelector.parameters.buildUpon()
    .setPreferredTextLanguage("en-US")
    .setSelectUndeterminedTextLanguage(true)
    .setPreferredTextRoleFlags(C.ROLE_FLAG_SUBTITLE)
    .build()
trackSelector.parameters = trackSelectorParams
player = ExoPlayer.Builder(context).setTrackSelector(trackSelector).build()

And loading the uri with the following code

fun setUri(uri: String) {
    val mediaItem = MediaItem.Builder()
        .setUri(uri)
        .setMimeType(MimeTypes.APPLICATION_MP4VTT)
        .build()
    player.setMediaItem(mediaItem)
}

Drive links for files: 2:13 video: https://drive.google.com/file/d/1II826CYfICX54-IlHth8Z291jLsB20E1/view?usp=sharing 2:13 video subtitles extracted with mp4box: https://drive.google.com/file/d/1oj72WF0IbiiV9YiZF5oQGqdAhsNtEtev/view?usp=sharing 2:30 video: https://drive.google.com/file/d/1DMmP7bTrI4CAB_3Znrc9CmCrQ0ko8e1y/view?usp=sharing 2:30 video subtitles extracted with mp4box: https://drive.google.com/file/d/1xgQzzfQV2CNeZcin-wDb6JBMFSU95zwJ/view?usp=sharing

I've tried loads of things to try and fix this behavior but nothing seems to improve it, any further guidance would be GREATLY appreciated.

[ojw28]

The problem is that all of the subtitle sample data is appended right at the very end of the file (e.g., at ~20.2MB for the 2:30 video). When the player is streaming this file, it will only get access to the subtitle samples to display them at the point when it's loaded the file right to the end. This is when they suddenly appear. This is also why you'll see them appear more quickly if you seek to a position that causes playback to start near to the end of the file, since loading from that point to the end requires loading less media than when starting from the beginning. You will find that the problem is more extreme the longer the video file, for the same reason.

The subtitle sample data should really be interleaved through the file, so that a subtitle sample is placed next to (or nearby) the corresponding audio and video samples. This means that a player will encounter each sample as it needs it during playback, as it does for audio and video samples.

[ojw28]

By the way, if you want to see where the samples are placed, this is an excellent (free) tool: https://gpac.github.io/mp4box.js/test/filereader.html

You can load one of your files, then go to "Sample View", and look at the offset column for different samples in different tracks. You'll find that for the wvtt track (which contains the subtitles), the offset are all right at the very end of the file. Where-as for the audio and video tracks they're interleaved.

[JustinGuruTech]

@ojw28 This is so helpful, thank you so much! Do you happen to have a tool recommendation for embedding vtt into mp4 in a way that will allow me to control this? We've been using a Mac app called Subler but that doesn't seem to have support for interleaving.

[ojw28]

I don't I'm afraid. It might be worth looking at FFmpeg and MP4Box, but I don't know to what extent they support this.

One final thing:

val mediaItem = MediaItem.Builder()
        .setUri(uri)
        .setMimeType(MimeTypes.APPLICATION_MP4VTT)
        .build()

The MIME type here is supposed to be the MIME type of the whole media file, so if it's set to anything it should be set to VIDEO_MP4 (it's fine / better just to not set it for this case). You probably set it having read some of our documentation around subtitles, but those examples use standalone subtitle media files, rather than MP4 files within which the subtitles are embedded.