Background Replacement Images are not scaling as per the video tile

[YumTheDeathGod]

Describe the bug In our react native application we are using base64 images on native android side for background replacement functionality. But in our case the image is getting upscaled and only a part of the image is being set to the background of the video tile. Please suggest. To set the image as a background I have performed the following steps: Calling native method through bridge: RNFetchBlob.fetch( "GET", "https://images.generated.photos/WFfsbGAPqoAQywfdZ0VDZEnXexq5eLoEiJyBdK_2PVM/rs:fit:256:256/czM6Ly9pY29uczgu/Z3Bob3Rvcy1wcm9k/LnBob3Rvcy92M18w/NjA1MzgxLmpwZw.jpg" ) .then((res) => { console.log(res); let status = res.info().status; console.log(status); if (status == 200) { let base64 = res.base64(); **NativeFunction.convertImageToBitmap( !isVoiceFocusEnabled, base64 );** } }) .catch((error) => { console.log("Error occured", error); });

I have converted base64(Which is passed from react native side iteself) formatted image to bitmap:

` @ReactMethod fun convertImageToBitmap(enabled: Boolean, base64String: String, screenWidth: Int, screenHeight: Int) {

    try {
        val decodedString = Base64.decode(base64String, Base64.DEFAULT)
        logger.info(TAG, "decode base 64 string is: ${decodedString}")
        val decodedByte = BitmapFactory.decodeByteArray(decodedString, 0, decodedString.size)
        replaceBackgroundImage(enabled, decodedByte)
    } catch (e: IOException) {
        logger.info(TAG, "Bitmap conversion failed ${e}")
    }

}`

3. After this conversion I have passed this image to BackgroundReplacementConfiguration: ` fun replaceBackgroundImage(enabled: Boolean, bitmapImage: Bitmap) { logger.info(TAG, "Background replacement functionality: ${enabled}") if(enabled) { val surfaceTextureCaptureSourceFactory = DefaultSurfaceTextureCaptureSourceFactory(logger, eglCoreFactory) val cameraCaptureSource= DefaultCameraCaptureSource(reactApplicationContext, logger, surfaceTextureCaptureSourceFactory)

       val backgroundReplacementVideoFrameProcessor = BackgroundReplacementVideoFrameProcessor(
           ConsoleLogger(LogLevel.DEBUG),
           eglCoreFactory,
           reactApplicationContext,
           BackgroundReplacementConfiguration(bitmapImage)
       )
   
       cameraCaptureSource.addVideoSink(backgroundReplacementVideoFrameProcessor)
       meetingSession?.audioVideo?.startLocalVideo(backgroundReplacementVideoFrameProcessor)
       cameraCaptureSource.start();
   } else {
       meetingSession?.audioVideo?.startLocalVideo()
   }

   } ` But unfortunately, I am not able to set it properly.

To Reproduce Steps to reproduce the behavior: Implement the react native demo project as per the guidelines from React Native Chime itself Implement background image replacement filter as per the documentation or you can use above code also. Join the meeting (There is no need to join from remote partcipant. you can see the image getting scaled on local side also)

Expected behavior Image should have scaled as per the video frame dimensions.

Test environment Info (please complete the following information):

• Device: Lenovo TB-8705F
• OS: Android 10
• Version amazon-chime-sdk: 0.17.0
• Version amazon-chime-sdk-media: : 0.17.0
• Version amazon-chime-sdk-media-machine-learning : 0.1.0

Additional context I cannot downscale or upscaled the image as per the device dimension. Please assist I have tried some kotlin approaches - BitmapFactory.Options().inSample image options and createScaledBitmap functions etc... none of them worked. Kindly, let me know if my approach is correct or not? and if not then how should I make update the code. Thank you

[hokyungh]

Thank you for opening issue. We'll look into it and update the ticket.

[richhx]

Thanks for pointing this out. I tested the image you gave against the changes in https://github.com/aws/amazon-chime-sdk-android/pull/459 and it should be fixed after the PR is merged. i.e. this appears to be the same issue as https://github.com/aws-samples/amazon-chime-react-native-demo/issues/145 where the scaling of the images was done incorrectly.

[YumTheDeathGod]

Hey @hokyungh , @richhx , Thanks for reply and support. Waiting for PR merge. If it is possible will you guys please provide a quick walk through where I can understand how can use simple image to set background of video tile without converting it into base64 or any other type except for bitmap from react native itself. It will be great help for me. Thank you guys for the efforts and help. Really appreciated

[richhx]

Should be merged into development branch now (master in the next release). Thanks for your patience.

w.r.t your question on converting to bitmap from solely react-native itself, I do not know of a way. The blur/replacement feature was designed for Android/iOS SDK whereas the react-native doesn't necessarily offer methods to convert to the mobile interfaces/classes from what I can see. The following link is a guide to get the bitmap from a URL in Android if that helps at all.

[YumTheDeathGod]

Thank you @richhx , I will take a look into it again. Thanks

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hokyungh]

This code has been released as of 0.17.2. Closing. Free free to reopen if you see any issue with latest.