MediaItems textures rendered not in the order they provided to Composition

[mikekudzin]

Version

Media3 1.3.1

More version details

-

Devices that reproduce the issue

Pixel 4a running Android 13 Pixel 6a running 14

Devices that do not reproduce the issue

No response

Reproducible in the demo app?

Not tested

Reproduction steps

I am trying to generate output video using following inputs:

1. Overlay with a transparent bitmap holding drawings, texts.
2. Video item transformed in some way (just rotation in the example)
3. Background which is a simple non-transparent PNG

Code is pretty straightforward

        val bitmapOverlayWithText = BitmapOverlay.createStaticBitmapOverlay(overlayBitmap)
        val overlayFrameEMI = EditedMediaItem.Builder(MediaItem.fromUri(frameFile.toUri()))
            .setFrameRate(D_F_RATE)
            .setDurationUs(D_DURATION)
            .setRemoveAudio(true)
            .setEffects(
                Effects(
                    listOf(),
                    listOf(
                        OverlayEffect(ImmutableList.of(bitmapOverlayWithText)),
                    )
                )
            )
            .build()

        val backgroundMI = MediaItem.fromUri(backgroundBitmapFile.toUri())
        val backgroundEMI = EditedMediaItem.Builder(backgroundMI)
            .setFrameRate(D_F_RATE)
            .setDurationUs(D_DURATION)
            .setRemoveAudio(true)
            .build()

        val videoMI = MediaItem.Builder().setUri(baseMediaUri)
            .setClippingConfiguration(
                MediaItem.ClippingConfiguration.Builder()
                    .setStartPositionUs(0)
                    .setEndPositionUs(D_DURATION)
                    .build()
            )
            .build()

        val videoEMI = EditedMediaItem.Builder(videoMI)
            .setEffects(
                Effects(
                    listOf(),
                    listOf(ScaleAndRotateTransformation.Builder().setRotationDegrees(45f).build())
                )
            )
            .build()

        val composition = Composition.Builder(
            EditedMediaItemSequence(overlayFrameEMI),
            EditedMediaItemSequence(videoEMI),
            EditedMediaItemSequence(backgroundEMI),            
        )
            .build()

        val transformer: Transformer =
            Transformer.Builder(context)
                .setVideoMimeType(MimeTypes.VIDEO_H264)
                .setAudioMimeType(MimeTypes.AUDIO_AAC)
                .build()
        transformer.start(composition, outputFilePath)

Output of the code above:

https://github.com/androidx/media/assets/7223881/72b8e185-c28b-423e-9ad0-9651bed6fe32

There is no original video in output. BUT SOMETIMES (rare) it produces expected output.

If I remove background from the composition

        val composition = Composition.Builder(
            EditedMediaItemSequence(overlayFrameEMI),
            EditedMediaItemSequence(videoEMI),
        )
            .build()

The output is nearly what I am expecting except the background (sic!)

https://github.com/androidx/media/assets/7223881/1ec220d9-0cdf-488c-a253-c8d3601239d4

And finally, setting alphaScale to 0.9

            ...
            .setVideoCompositorSettings(object : VideoCompositorSettings {
                override fun getOutputSize(inputSizes: MutableList<Size>): Size {
                    return inputSizes[0]
                }

                override fun getOverlaySettings(inputId: Int, presentationTimeUs: Long): OverlaySettings {
                    return OverlaySettings.Builder()
                        .setAlphaScale(0.9f)
                        .build()
                }
            })
            .build()

https://github.com/androidx/media/assets/7223881/7f77c769-42fe-46b6-979d-e78fd1d410c9

It looks like the video EditedMediaItemSequence get's overlaid by background EditedMediaItemSequence at some point.

Documentation for DefaultVideoCompositor states:

When composited, textures are overlaid over one another in the reverse order of their registration order, so that the first registered source is on the very top. The way the textures are overlaid can be customized using the {@link OverlaySettings} output by {@link VideoCompositorSettings}.

I can be mistaken here, though, since Composition class itself doesn't provide explicit statement about rendering order. If so, what API/approach I can use to force rendering order?

Expected result

Textures are drawn in the reverse order they provided to Composition

Actual result

Order is not preserved, some medias got overlapped. Unpredictable output.

Media

-

Bug Report

• [ ] You will email the zip file produced by adb bugreport to android-media-github@google.com after filing this issue.

[droid-girl]

Hi @mikekudzin , The issue #1029 with ordering of EditedMediaItems was fixed in this commit and it is a part of the next 1.4.0 release. This should provide a fix for the bug you see.

I quickly looked through the code you have provided. You can also use TextOverlay as an effect and apply it to your video. This might be another approach on how to do it.

[mikekudzin]

Thank you! Switching to 1.4.0-beta1 containing #1029 fixed the issue.