search

androidx / media

Jetpack Media3 support libraries for media use cases, including ExoPlayer, an extensible media player for Android
https://developer.android.com/media/media3
Apache License 2.0
1.74k stars 416 forks source link

Transformer: multi-asset video composition #1919

Open chenru1106 opened 3 days ago

chenru1106 commented 3 days ago

Hello team： I have two questions. How can we adjust the hierarchy of multiple videos and support the second video to start playing at the specified time of the first video

droid-girl commented 3 days ago

Hi @chenru1106 , Could you provide more details? Are you trying to build picture-in-picture kind of composition where the secondary video appears not from the start, but for example after 5 seconds of the start of the primary video?

chenru1106 commented 3 days ago

Hi @chenru1106 , Could you provide more details? Are you trying to build picture-in-picture kind of composition where the secondary video appears not from the start, but for example after 5 seconds of the start of the primary video?

yes

chenru1106 commented 2 days ago

I added a picture in picture in the demo. The video is 1920 * 1080 and lasts for 20 minutes. 

 Composition.Builder compositionBuilder =
        new Composition.Builder(
            new EditedMediaItemSequence.Builder(editedMediaItemBuilder.build()).build(),
            new EditedMediaItemSequence.Builder(editedMediaItemBuilder.build()).build(),
            new EditedMediaItemSequence.Builder(editedMediaItemBuilder.build()).build(),
            new EditedMediaItemSequence.Builder(editedMediaItemBuilder.build()).build()
        );
 compositionBuilder
          .setVideoCompositorSettings(new VideoCompositorSettings() {
            @Override
            public Size getOutputSize(List<Size> inputSizes) {
              return inputSizes.get(0);
            }

            @Override
            public OverlaySettings getOverlaySettings(int inputId, long presentationTimeUs) {
              if (inputId != 0) {
               return new StaticOverlaySettings.Builder().setAlphaScale(0.5f)
                   .setScale(0.5f,0.5f)
                    .setBackgroundFrameAnchor(-0.9f, -0.9f).build();
              }
              return new StaticOverlaySettings.Builder().build();
            }
          }));

There are two questions here

  1. The video with added transparency is hidden in the lower layer of the main video
  2. At around 90% progress, an exception will be thrown 
    Export error
androidx.media3.transformer.ExportException: Video frame processing error
at androidx.media3.transformer.VideoSampleExporter$VideoGraphWrapper.onError(VideoSampleExporter.java:541)
at androidx.media3.effect.MultipleInputVideoGraph.lambda$handleVideoFrameProcessingException$2$androidx-media3-effect-MultipleInputVideoGraph(MultipleInputVideoGraph.java:441)
at androidx.media3.effect.MultipleInputVideoGraph$$ExternalSyntheticLambda0.run(D8$$SyntheticClass:0)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at androidx.media3.effect.MultipleInputVideoGraph.handleVideoFrameProcessingException(MultipleInputVideoGraph.java:439)
at androidx.media3.effect.MultipleInputVideoGraph.access$500(MultipleInputVideoGraph.java:66)
at androidx.media3.effect.MultipleInputVideoGraph$3.onError(MultipleInputVideoGraph.java:259)
at androidx.media3.effect.FinalShaderProgramWrapper.lambda$renderFrame$5$androidx-media3-effect-FinalShaderProgramWrapper(FinalShaderProgramWrapper.java:422)
at androidx.media3.effect.FinalShaderProgramWrapper$$ExternalSyntheticLambda0.run(D8$$SyntheticClass:0)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at androidx.media3.effect.FinalShaderProgramWrapper.renderFrame(FinalShaderProgramWrapper.java:420)
at androidx.media3.effect.FinalShaderProgramWrapper.queueInputFrame(FinalShaderProgramWrapper.java:240)
at androidx.media3.effect.FrameConsumptionManager.lambda$onReadyToAcceptInputFrame$0$androidx-media3-effect-FrameConsumptionManager(FrameConsumptionManager.java:74)
at androidx.media3.effect.FrameConsumptionManager$$ExternalSyntheticLambda2.run(D8$$SyntheticClass:0)
at androidx.media3.effect.VideoFrameProcessingTaskExecutor.lambda$wrapTaskAndSubmitToExecutorService$3$androidx-media3-effect-VideoFrameProcessingTaskExecutor(VideoFrameProcessingTaskExecutor.java:290)
at androidx.media3.effect.VideoFrameProcessingTaskExecutor$$ExternalSyntheticLambda0.run(D8$$SyntheticClass:0)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:487)
at java.util.concurrent.FutureTask.run(FutureTask.java:290)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:307)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
at java.lang.Thread.run(Thread.java:1012)
Caused by: androidx.media3.common.VideoFrameProcessingException: androidx.media3.common.util.GlUtil$GlException: glError: out of memory
at androidx.media3.effect.FinalShaderProgramWrapper.lambda$renderFrame$5$androidx-media3-effect-FinalShaderProgramWrapper(FinalShaderProgramWrapper.java:423)
at androidx.media3.effect.FinalShaderProgramWrapper$$ExternalSyntheticLambda0.run(D8$$SyntheticClass:0) 
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31) 
at androidx.media3.effect.FinalShaderProgramWrapper.renderFrame(FinalShaderProgramWrapper.java:420) 
at androidx.media3.effect.FinalShaderProgramWrapper.queueInputFrame(FinalShaderProgramWrapper.java:240) 
at androidx.media3.effect.FrameConsumptionManager.lambda$onReadyToAcceptInputFrame$0$androidx-media3-effect-FrameConsumptionManager(FrameConsumptionManager.java:74) 
at androidx.media3.effect.FrameConsumptionManager$$ExternalSyntheticLambda2.run(D8$$SyntheticClass:0) 
at androidx.media3.effect.VideoFrameProcessingTaskExecutor.lambda$wrapTaskAndSubmitToExecutorService$3$androidx-media3-effect-VideoFrameProcessingTaskExecutor(VideoFrameProcessingTaskExecutor.java:290) 
at androidx.media3.effect.VideoFrameProcessingTaskExecutor$$ExternalSyntheticLambda0.run(D8$$SyntheticClass:0) 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:487) 
at java.util.concurrent.FutureTask.run(FutureTask.java:290) 
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:307) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644) 
at java.lang.Thread.run(Thread.java:1012) 
Caused by: androidx.media3.common.util.GlUtil$GlException: glError: out of memory
at androidx.media3.common.util.GlUtil.checkGlError(GlUtil.java:515)
at androidx.media3.common.util.GlUtil.createGlSyncFence(GlUtil.java:447)
at androidx.media3.effect.FinalShaderProgramWrapper.renderFrameToOutputTexture(FinalShaderProgramWrapper.java:471)
at androidx.media3.effect.FinalShaderProgramWrapper.renderFrame(FinalShaderProgramWrapper.java:417)
at androidx.media3.effect.FinalShaderProgramWrapper.queueInputFrame(FinalShaderProgramWrapper.java:240) 
at androidx.media3.effect.FrameConsumptionManager.lambda$onReadyToAcceptInputFrame$0$androidx-media3-effect-FrameConsumptionManager(FrameConsumptionManager.java:74) 
at androidx.media3.effect.FrameConsumptionManager$$ExternalSyntheticLambda2.run(D8$$SyntheticClass:0) 
at androidx.media3.effect.VideoFrameProcessingTaskExecutor.lambda$wrapTaskAndSubmitToExecutorService$3$androidx-media3-effect-VideoFrameProcessingTaskExecutor(VideoFrameProcessingTaskExecutor.java:290) 
at androidx.media3.effect.VideoFrameProcessingTaskExecutor$$ExternalSyntheticLambda0.run(D8$$SyntheticClass:0) 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:487) 
at java.util.concurrent.FutureTask.run(FutureTask.java:290) 
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:307) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644) 
at java.lang.Thread.run(Thread.java:1012) 
droid-girl commented 2 days ago

@chenru1106 thanks for reporting and the logs. While this functionality is still work in progress and picture in picture should be working, the delay you want to implement is not yet supported. Is your secondary video longer in duration than the primary video (the first sequence)?

Assigning to @claincly to look into it

claincly commented 2 days ago

In multi-video composition, we don't currently support starting videos from at different times - so delayed start is not supported. We have this in our backlog though.

Apart from that, the error shows glError: out of memory, I wonder what GL processing are you doing on your end? Is the device under load? Do you have your own effects running? The default pipeline should be fairly conservative on memory usage and should not cause OOM in general.

chenru1106 commented 2 days ago

@chenru1106 thanks for reporting and the logs. While this functionality is still work in progress and picture in picture should be working, the delay you want to implement is not yet supported. Is your secondary video longer in duration than the primary video (the first sequence)?

Assigning to @claincly to look into it

Will not exceed the duration of the primary video

chenru1106 commented 2 days ago

In multi-video composition, we don't currently support starting videos from at different times - so delayed start is not supported. We have this in our backlog though.

Apart from that, the error shows glError: out of memory, I wonder what GL processing are you doing on your end? Is the device under load? Do you have your own effects running? The default pipeline should be fairly conservative on memory usage and should not cause OOM in general.

image The logic I added directly in the TransormerActivity creatComposition method of the demo

chenru1106 commented 1 day ago

Why is the composition set like this? The primary video is in the upper layer

 new EditedMediaItemSequence.Builder(editedMediaItemBuilder.build()).build(),  //primary video
  new EditedMediaItemSequence.Builder(editedMediaItemBuilder.build()).build(),
claincly commented 1 day ago

Why is the composition set like this? The primary video is in the upper layer

I don't understand your question. What do you want to achieve?