Colours discrepancy when applying overlay

[mikekudzin]

Version

Media3 pre-release (alpha, beta or RC not in this list)

More version details

1.4.0-beta01

Devices that reproduce the issue

Pixel 4a Android 13

Devices that do not reproduce the issue

No response

Reproducible in the demo app?

Not tested

Reproduction steps

Overlaying video with a bitmap.

no. 1: Create transparent media items with a BitmapOverlay Code:

    val bitmapOverlayWithText = BitmapOverlay.createStaticBitmapOverlay(overlayBitmap)
    val overlayFrameEMI = EditedMediaItem.Builder(MediaItem.fromUri(frameFile.toUri()))
        .setFrameRate(D_F_RATE)
        .setDurationUs(D_DURATION)
        .setRemoveAudio(true)
        .setEffects(
            Effects(
                listOf(),
                listOf(
                    OverlayEffect(ImmutableList.of(bitmapOverlayWithText)),
                )
            )
        )
        .build()

    val backgroundMI = MediaItem.fromUri(backgroundBitmapFile.toUri())
    val backgroundEMI = EditedMediaItem.Builder(backgroundMI)
        .setFrameRate(D_F_RATE)
        .setDurationUs(D_DURATION)
        .setRemoveAudio(true)
        .build()

    val videoMI = MediaItem.Builder().setUri(baseMediaUri)
        .setClippingConfiguration(
            MediaItem.ClippingConfiguration.Builder()
                .setStartPositionUs(0)
                .setEndPositionUs(D_DURATION)
                .build()
        )
        .build()

    val videoEMI = EditedMediaItem.Builder(videoMI)
        .setEffects(
            Effects(
                listOf(),
                listOf(ScaleAndRotateTransformation.Builder().setRotationDegrees(45f).build())
            )
        )
        .build()

    val composition = Composition.Builder(
        EditedMediaItemSequence(overlayFrameEMI),
        EditedMediaItemSequence(videoEMI),
        EditedMediaItemSequence(backgroundEMI),            
    )
        .build()

    val transformer: Transformer =
        Transformer.Builder(context)
            .setVideoMimeType(MimeTypes.VIDEO_H264)
            .setAudioMimeType(MimeTypes.AUDIO_AAC)
            .build()
    transformer.start(composition, outputFilePath)

Result:

no. 2: create MediaItem directly from the file Code diff val overlayFrameEMI = EditedMediaItem.Builder(MediaItem.fromUri(overlayFile.toUri())) .setFrameRate(D_F_RATE) .setDurationUs(D_DURATION) .build()

Although, the the last one looks close to origin, there is a significant colours discrepancy between the origin no. 1 and no. 2 I didn't observe no. 1 issue on 1.3.1

Expected result

Colours in the output are the same as in the origin.

Actual result

There is a difference between colours in the origin and BitmapOverlay

Media

-

Bug Report

• [X] You will email the zip file produced by adb bugreport to android-media-github@google.com after filing this issue.

[tof-tof]

Hi @mikekudzin, thanks for writing in.

just to make sure I understand the setup, are you saying in (1) you used overlays and in (2) you didn't use overlays at all?

Since you didn't see it in 1.3.1 I figure it could have some involvment to with the way we process color changing. Using the code in https://github.com/androidx/media/issues/1050#issuecomment-2132541344 may help your case

In addition I see there are some artefacts around the borders of your image. @ychaparov does any of the recent work you've done around improving sampling help with this?

[ychaparov]

Not really, we haven't made improvements to resampling of 45-degree rotated videos yet.

The original image bitmap overlay is not aligned with the no. 1 output -- the text is a bit scaled and rotated even though both images have resolution 1080x1920. I don't think BitmapOverlay causes this image modification.

@mikekudzin can you share your bitmap-reading code? How did you create overlayBitmap used in

    val bitmapOverlayWithText = BitmapOverlay.createStaticBitmapOverlay(overlayBitmap)

[yuriikonovalov]

@tof-tof @ychaparov I have the same problem. The version is 1.4.0-beta01. The device I'm using is Samsung S21. I've tried the suggestion in the #1050 comment but it does not help.

Here is the input and output resources:

• This image is added as a MediaItem

  

• This image is added as a BitmapOverlay

  

• This is a frame in an output video

  

  Here is the way the overlay bitmap is created:

   val stream = context.contentResolver.openInputStream(uri)
   val bitmap = BitmapFactory.decodeStream(stream)

[ychaparov]

you should probably set some bitmap factory options like this:

    if (Util.SDK_INT >= 26) {
      options = new BitmapFactory.Options();
      options.inPreferredColorSpace = ColorSpace.get(ColorSpace.Named.SRGB);
    }

Here's how the MediaItem image BitmapFactory options will be set https://github.com/androidx/media/blob/d833d59124d795afc146322fe488b2c0d4b9af6a/libraries/transformer/src/main/java/androidx/media3/transformer/DefaultAssetLoaderFactory.java#L82-L86

[yuriikonovalov]

@ychaparov just tried that and it stays the same as I mentioned above.

[mikekudzin]

just to make sure I understand the setup, are you saying in (1) you used overlays and in (2) you didn't use overlays at all?

@tof-tof Correct.

@ychaparov here is bitmap initialisation code.

    private fun createCanvasBitmap(width: Int = D_WIDTH, height: Int = D_HEIGHT): Bitmap {
        val bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
        return bitmap
    }

    protected fun initCanvas(width: Int = D_WIDTH, height: Int = D_HEIGHT): Pair<Canvas, Bitmap> {
        val bitmap = createCanvasBitmap(width, height)
        val canvas = Canvas(bitmap)
        return canvas to bitmap
    }

I also tried approach with the BitmapFactory

        val options = BitmapFactory.Options()
        options.inPreferredColorSpace = ColorSpace.get(ColorSpace.Named.SRGB)
        val overlayBitmap = BitmapFactory.decodeFile(overlayFile.path, options)

Still observe the issue .

[droid-girl]

@yuriikonovalov and @mikekudzin how many EditedMediaItemSequences do you have in your composition?

[yuriikonovalov]

@droid-girl In my case, it's 3 EditedMediaItemSequences. One is for a video. Another one is a transparent image with those bitmap overlays. One more is just used to add the "a media item" image as a media item to the composition, not as a bitmap overlay so that the difference is visible.

[mikekudzin]

@droid-girl I use 3 EditedMediaItemSequences: background, video with some transformations, overlay

[droid-girl]

@ychaparov figured out the problem, we will have a fix soon in main. The issue was with Composition that contains more than 1 sequence containing video/image assets.

[ychaparov]

Could you please try verify if https://github.com/androidx/media/commit/7103f21da9498ec4208d5ed8fe49be27cd104986 fixes the issue for your use cases?

[yuriikonovalov]

@ychaparov Just tried that, it's fixed for my case. Thanks.

Just need to set SdrWorkingColorSpace. Otherwise colors will still be a little bit off but not like it was before.

Transformer.Builder(this)
    .setVideoFrameProcessorFactory(
        DefaultVideoFrameProcessor.Factory.Builder()
            .setSdrWorkingColorSpace(
                DefaultVideoFrameProcessor.WORKING_COLOR_SPACE_ORIGINAL
            )
            .build()
    )

With setting SdrWorkingColorSpace

Without setting SdrWorkingColorSpace - here it's possible to see some difference

[google-oss-bot]

Hey @mikekudzin. We need more information to resolve this issue but there hasn't been an update in 14 weekdays. I'm marking the issue as stale and if there are no new updates in the next 7 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[google-oss-bot]

Since there haven't been any recent updates here, I am going to close this issue.

@mikekudzin if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.