Open xxoo opened 1 month ago
The
SubtitleView
is not an option cause off screen rendering is required in this case.
Reading slightly between the lines here, are you trying to render subtitles as part of a batch transformation of the video?
I think the Media3 Transformer library would be the right tool for that (maybe you're already using it) - but I don't think it currently has support for rendering Cue
objects.
If I've understood the request correctly we can use this issue to track it as a potential enhancement.
@icbaker
Thanks for reply
Yes. I need to draw the video and selected subtitles into separate SurfaceTexture
s, and then process them further in subsequent steps. After that, blend them and complete the screen rendering. I found SubtitlePainter
and part of CanvasSubtitleOutput
are very helpful. However they are private apis. Perhaps the best way for now is to just copy them into my project and make the necessary changes.
Transformer APIs will be an option here. We do not support for rendering Cue
objects at the moment.
You can look into Transformer and Effect module. In particular, TextOverlay
. TextOverlay
does not support dynamic text at the moment but you can implement your custom implementation based on the existing one.
@droid-girl Thank you for the tip, I tried using BitmapOverlay
but seems it doesn't work with setVideoSurface()
.
My code looks like this
override fun onCues(cueGroup: CueGroup) {
if (cueGroup.cues.isEmpty() || exoPlayer.videoSize.width == 0 || exoPlayer.videoSize.height == 0) {
exoPlayer.setVideoEffects(emptyList())
} else {
val overlaysBuilder = ImmutableList.Builder<TextureOverlay>()
val bitmap = Bitmap.createBitmap(exoPlayer.videoSize.width, exoPlayer.videoSize.height, Bitmap.Config.ARGB_8888)
val canvas = Canvas(bitmap)
for (cue in cueGroup.cues) {
subtitlePainter.draw(cue, canvas)
}
val textOverlay = BitmapOverlay.createStaticBitmapOverlay(bitmap)
overlaysBuilder.add(textOverlay)
val overlays = overlaysBuilder.build()
val overlay = OverlayEffect(overlays)
exoPlayer.setVideoEffects(listOf(overlay))
}
}
@claincly could you help here?
I'm not familiar with the Cue object format, but one potential issue with the provided code is the size (exoPlayer.videoSize.width
), as it might not be what you'd expect when using Effects.
I'm a bit confused - you wanted to show the text subtitles when playing the video right? I wonder if the following would work:
We for example have a simple effect that imprints video timestamps onto the screen:
@claincly Thanks for the guidelines. The subtitle frame should be the same as the video frame. Cause the text can be shown in any part of the screen according to the Cue
properties. But I'm still not sure why the following error keeps showing if setVideoEffects
is called before prepare
E Playback error
androidx.media3.exoplayer.ExoPlaybackException: MediaCodecVideoRenderer error, index=0, format=Format(0, null, null, video/avc, avc1.64001F, -1, null, [848, 480, -1.0, ColorInfo(Unset color space, Unset color range, Unset color transfer, false, 8bit Luma, 8bit Chroma)], [-1, -1]), format_supported=YES
at androidx.media3.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:640)
at android.os.Handler.dispatchMessage(Handler.java:103)
at android.os.Looper.loopOnce(Looper.java:232)
at android.os.Looper.loop(Looper.java:317)
at android.os.HandlerThread.run(HandlerThread.java:85)
Caused by: androidx.media3.exoplayer.video.VideoSink$VideoSinkException: androidx.media3.common.VideoFrameProcessingException: java.lang.IllegalStateException: No call to setSamplerTexId() before bind.
at androidx.media3.exoplayer.video.CompositingVideoSinkProvider$VideoSinkImpl.lambda$onError$3$androidx-media3-exoplayer-video-CompositingVideoSinkProvider$VideoSinkImpl(CompositingVideoSinkProvider.java:879)
at androidx.media3.exoplayer.video.CompositingVideoSinkProvider$VideoSinkImpl$$ExternalSyntheticLambda2.run(D8$$SyntheticClass:0)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at androidx.media3.exoplayer.video.CompositingVideoSinkProvider$VideoSinkImpl.onError(CompositingVideoSinkProvider.java:874)
at androidx.media3.exoplayer.video.CompositingVideoSinkProvider.onError(CompositingVideoSinkProvider.java:349)
at androidx.media3.effect.SingleInputVideoGraph$1.lambda$onError$2$androidx-media3-effect-SingleInputVideoGraph$1(SingleInputVideoGraph.java:148)
at androidx.media3.effect.SingleInputVideoGraph$1$$ExternalSyntheticLambda3.run(D8$$SyntheticClass:0)
at android.os.Handler.handleCallback(Handler.java:959)
at android.os.Handler.dispatchMessage(Handler.java:100)
... 3 more
Caused by: androidx.media3.common.VideoFrameProcessingException: java.lang.IllegalStateException: No call to setSamplerTexId() before bind.
at androidx.media3.effect.VideoFrameProcessingTaskExecutor.handleException(VideoFrameProcessingTaskExecutor.java:222)
at androidx.media3.effect.VideoFrameProcessingTaskExecutor.lambda$wrapTaskAndSubmitToExecutorService$2$androidx-media3-effect-VideoFrameProcessingTaskExecutor(VideoFrameProcessingTaskExecutor.java:208)
at androidx.media3.effect.VideoFrameProcessingTaskExecutor$$ExternalSyntheticLambda0.run(D8$$SyntheticClass:0)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:487)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
at java.lang.Thread.run(Thread.java:1012)
Caused by: java.lang.IllegalStateException: No call to setSamplerTexId() before bind.
at androidx.media3.common.util.GlProgram$Uniform.bind(GlProgram.java:467)
at androidx.media3.common.util.GlProgram.bindAttributesAndUniforms(GlProgram.java:224)
at androidx.media3.effect.OverlayShaderProgram.drawFrame(OverlayShaderProgram.java:176)
at androidx.media3.effect.BaseGlShaderProgram.queueInputFrame(BaseGlShaderProgram.java:156)
at androidx.media3.effect.FrameConsumptionManager.lambda$queueInputFrame$1$androidx-media3-effect-FrameConsumptionManager(FrameConsumptionManager.java:96)
at androidx.media3.effect.FrameConsumptionManager$$ExternalSyntheticLambda1.run(D8$$SyntheticClass:0)
at androidx.media3.effect.VideoFrameProcessingTaskExecutor.lambda$wrapTaskAndSubmitToExecutorService$2$androidx-media3-effect-VideoFrameProcessingTaskExecutor(VideoFrameProcessingTaskExecutor.java:206)
... 6 more
subTitleFrame
is a Bitmap
instance that always with the current subtitle content. The subtitleId
updates with the bitmap content
exoPlayer.setVideoEffects(listOf(OverlayEffect(ImmutableList.of(object: BitmapOverlay() {
override fun getBitmap(presentationTimeUs: Long): Bitmap {
return subTitleFrame
}
override fun getTextureSize(presentationTimeUs: Long): Size {
return Size(subTitleFrame.width, subTitleFrame.height)
}
override fun getTextureId(presentationTimeUs: Long): Int {
return subtitleId
}
override fun configure(videoSize: Size) {
val old = subTitleFrame
subTitleFrame = Bitmap.createScaledBitmap(old, videoSize.width, videoSize.height, true)
old.recycle()
}
}))))
I'm not 100% familiar with the overlay part of the code base, but roughly looking at it I think most of our implementations don't need to implement the getTextureId()
method, are you doing some customization with the GL component?
OTOH I found this class that I think could be very similar to what you need?
Since
androidx.media3.ui.SubtitlePainter
is private in the package. I wonder if there is an alternate path to achieve this? TheSubtitleView
is not an option cause off screen rendering is required in this case.