Render subtitles (Cue objects) via an Effect

[xxoo]

Since androidx.media3.ui.SubtitlePainter is private in the package. I wonder if there is an alternate path to achieve this? The SubtitleView is not an option cause off screen rendering is required in this case.

[icbaker]

The SubtitleView is not an option cause off screen rendering is required in this case.

Reading slightly between the lines here, are you trying to render subtitles as part of a batch transformation of the video?

I think the Media3 Transformer library would be the right tool for that (maybe you're already using it) - but I don't think it currently has support for rendering Cue objects.

If I've understood the request correctly we can use this issue to track it as a potential enhancement.

[xxoo]

@icbaker Thanks for reply Yes. I need to draw the video and selected subtitles into separate SurfaceTextures, and then process them further in subsequent steps. After that, blend them and complete the screen rendering. I found SubtitlePainter and part of CanvasSubtitleOutput are very helpful. However they are private apis. Perhaps the best way for now is to just copy them into my project and make the necessary changes.

[droid-girl]

Transformer APIs will be an option here. We do not support for rendering Cue objects at the moment. You can look into Transformer and Effect module. In particular, TextOverlay. TextOverlay does not support dynamic text at the moment but you can implement your custom implementation based on the existing one.

[xxoo]

@droid-girl Thank you for the tip, I tried using BitmapOverlay but seems it doesn't work with setVideoSurface(). My code looks like this

override fun onCues(cueGroup: CueGroup) {
    if (cueGroup.cues.isEmpty() || exoPlayer.videoSize.width == 0 || exoPlayer.videoSize.height == 0) {
        exoPlayer.setVideoEffects(emptyList())
    } else {
        val overlaysBuilder = ImmutableList.Builder<TextureOverlay>()
        val bitmap = Bitmap.createBitmap(exoPlayer.videoSize.width, exoPlayer.videoSize.height, Bitmap.Config.ARGB_8888)
        val canvas = Canvas(bitmap)
        for (cue in cueGroup.cues) {
            subtitlePainter.draw(cue, canvas)
        }
        val textOverlay = BitmapOverlay.createStaticBitmapOverlay(bitmap)
        overlaysBuilder.add(textOverlay)
        val overlays = overlaysBuilder.build()
        val overlay = OverlayEffect(overlays)
        exoPlayer.setVideoEffects(listOf(overlay))
    }
}

[droid-girl]

@claincly could you help here?

[claincly]

I'm not familiar with the Cue object format, but one potential issue with the provided code is the size (exoPlayer.videoSize.width), as it might not be what you'd expect when using Effects.

I'm a bit confused - you wanted to show the text subtitles when playing the video right? I wonder if the following would work:

• Implement a text overlay / bitmap overlay that receives input timestamps
• Match the input timestamps with the Cue objects when they are available
• Draw the corresponding subtitle text

We for example have a simple effect that imprints video timestamps onto the screen:

https://github.com/androidx/media/blob/b01c6ffcb3fca3d038476dab5d3bc9c9f2010781/libraries/transformer/src/androidTest/java/androidx/media3/transformer/mh/performance/PlaybackTestUtil.java#L35-L61

[xxoo]

@claincly Thanks for the guidelines. The subtitle frame should be the same as the video frame. Cause the text can be shown in any part of the screen according to the Cue properties. But I'm still not sure why the following error keeps showing if setVideoEffects is called before prepare

 E  Playback error
androidx.media3.exoplayer.ExoPlaybackException: MediaCodecVideoRenderer error, index=0, format=Format(0, null, null, video/avc, avc1.64001F, -1, null, [848, 480, -1.0, ColorInfo(Unset color space, Unset color range, Unset color transfer, false, 8bit Luma, 8bit Chroma)], [-1, -1]), format_supported=YES
    at androidx.media3.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:640)
    at android.os.Handler.dispatchMessage(Handler.java:103)
    at android.os.Looper.loopOnce(Looper.java:232)
    at android.os.Looper.loop(Looper.java:317)
    at android.os.HandlerThread.run(HandlerThread.java:85)
Caused by: androidx.media3.exoplayer.video.VideoSink$VideoSinkException: androidx.media3.common.VideoFrameProcessingException: java.lang.IllegalStateException: No call to setSamplerTexId() before bind.
    at androidx.media3.exoplayer.video.CompositingVideoSinkProvider$VideoSinkImpl.lambda$onError$3$androidx-media3-exoplayer-video-CompositingVideoSinkProvider$VideoSinkImpl(CompositingVideoSinkProvider.java:879)
    at androidx.media3.exoplayer.video.CompositingVideoSinkProvider$VideoSinkImpl$$ExternalSyntheticLambda2.run(D8$$SyntheticClass:0)
    at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
    at androidx.media3.exoplayer.video.CompositingVideoSinkProvider$VideoSinkImpl.onError(CompositingVideoSinkProvider.java:874)
    at androidx.media3.exoplayer.video.CompositingVideoSinkProvider.onError(CompositingVideoSinkProvider.java:349)
    at androidx.media3.effect.SingleInputVideoGraph$1.lambda$onError$2$androidx-media3-effect-SingleInputVideoGraph$1(SingleInputVideoGraph.java:148)
    at androidx.media3.effect.SingleInputVideoGraph$1$$ExternalSyntheticLambda3.run(D8$$SyntheticClass:0)
    at android.os.Handler.handleCallback(Handler.java:959)
    at android.os.Handler.dispatchMessage(Handler.java:100)
    ... 3 more
Caused by: androidx.media3.common.VideoFrameProcessingException: java.lang.IllegalStateException: No call to setSamplerTexId() before bind.
    at androidx.media3.effect.VideoFrameProcessingTaskExecutor.handleException(VideoFrameProcessingTaskExecutor.java:222)
    at androidx.media3.effect.VideoFrameProcessingTaskExecutor.lambda$wrapTaskAndSubmitToExecutorService$2$androidx-media3-effect-VideoFrameProcessingTaskExecutor(VideoFrameProcessingTaskExecutor.java:208)
    at androidx.media3.effect.VideoFrameProcessingTaskExecutor$$ExternalSyntheticLambda0.run(D8$$SyntheticClass:0)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:487)
    at java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
    at java.lang.Thread.run(Thread.java:1012)
Caused by: java.lang.IllegalStateException: No call to setSamplerTexId() before bind.
    at androidx.media3.common.util.GlProgram$Uniform.bind(GlProgram.java:467)
    at androidx.media3.common.util.GlProgram.bindAttributesAndUniforms(GlProgram.java:224)
    at androidx.media3.effect.OverlayShaderProgram.drawFrame(OverlayShaderProgram.java:176)
    at androidx.media3.effect.BaseGlShaderProgram.queueInputFrame(BaseGlShaderProgram.java:156)
    at androidx.media3.effect.FrameConsumptionManager.lambda$queueInputFrame$1$androidx-media3-effect-FrameConsumptionManager(FrameConsumptionManager.java:96)
    at androidx.media3.effect.FrameConsumptionManager$$ExternalSyntheticLambda1.run(D8$$SyntheticClass:0)
    at androidx.media3.effect.VideoFrameProcessingTaskExecutor.lambda$wrapTaskAndSubmitToExecutorService$2$androidx-media3-effect-VideoFrameProcessingTaskExecutor(VideoFrameProcessingTaskExecutor.java:206)
    ... 6 more

subTitleFrame is a Bitmap instance that always with the current subtitle content. The subtitleId updates with the bitmap content

exoPlayer.setVideoEffects(listOf(OverlayEffect(ImmutableList.of(object: BitmapOverlay() {
    override fun getBitmap(presentationTimeUs: Long): Bitmap {
        return subTitleFrame
    }
    override fun getTextureSize(presentationTimeUs: Long): Size {
        return Size(subTitleFrame.width, subTitleFrame.height)
    }
    override fun getTextureId(presentationTimeUs: Long): Int {
        return subtitleId
    }
    override fun configure(videoSize: Size) {
        val old = subTitleFrame
        subTitleFrame = Bitmap.createScaledBitmap(old, videoSize.width, videoSize.height, true)
        old.recycle()
    }
}))))

[claincly]

I'm not 100% familiar with the overlay part of the code base, but roughly looking at it I think most of our implementations don't need to implement the getTextureId() method, are you doing some customization with the GL component?

OTOH I found this class that I think could be very similar to what you need?

https://github.com/androidx/media/blob/release/libraries/effect/src/main/java/androidx/media3/effect/DrawableOverlay.java